Databricks SME

Key Responsibilities: Architecture & Platform Design • Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework • Define reference architectures for batch, streaming, analytics, and ML workloads • Select and standardize cluster, compute, and workspace architectures • Design multi-workspace strategies (dev/test/prod, shared vs. isolated) • Ensure architectures meet scalability, availability, and performance requirements Well-Architected Framework Alignment Apply Databricks best practices across all pillars, including: • Security & Governance (Unity Catalog, IAM, data access controls) • Reliability & Resilience (job retries, checkpointing, failure isolation) • Performance Efficiency (cluster sizing, autoscaling, caching) • Cost Optimization (compute policies, workload separation, monitoring) • Operational Excellence (monitoring, automation, CI/CD, runbooks) Implementation & Engineering • Lead Databricks workspace, cluster, and Unity Catalog implementations • Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming • Build and optimize ETL/ELT pipelines using Spark and SQL • Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking) • Establish CI/CD pipelines for notebooks, jobs, and infrastructure Security, Governance & Compliance • Implement Unity Catalog for centralized governance • Define data classification, lineage, and audit strategies • Enforce least-privilege access and secure networking patterns • Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable) Operations & Optimization • Monitor platform health, performance, and cost • Troubleshoot production issues across jobs, clusters, and data pipelines • Perform workload tuning and cost-performance optimization • Define SLOs, alerts, and operational metrics Collaboration & Leadership • Partner with Data Engineering, Analytics, ML, Platform, and Security teams • Translate business requirements into technical architectures • Provide architectural guidance and technical mentorship • Communicate risks, tradeoffs, and recommendations to leadership Required Qualifications: Experience • 7+ years in data engineering, analytics, or platform architecture • 3–5+ years hands-on Databricks experience in production environments • Proven experience applying the Databricks Well-Architected Framework • Experience designing cloud-native lakehouse architectures • Experience supporting mission-critical data platforms Technical Skills • Databricks Lakehouse Platform • Apache Spark (PySpark / Scala / Spark SQL) • Delta Lake, Delta Live Tables, Structured Streaming • Unity Catalog (governance, lineage, access controls) • Cloud platforms: AWS, Azure, or GCP • Infrastructure as Code (Terraform strongly preferred) • CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.) • Data formats and protocols (Parquet, JSON, Avro) Certifications Required: • Databricks Certified Data Engineer Professional • Databricks Certified Professional Architect (or equivalent advanced certification) Preferred / Additional Certifications • AWS Certified Solutions Architect (Associate or Professional) • Azure Solutions Architect Expert • Google Professional Data Engineer • Databricks Machine Learning Professional • Snowflake or other cloud data platform certifications Soft Skills • Strong architectural decision-making and documentation skills • Excellent communication with technical and non-technical stakeholders • Ability to lead design reviews and architecture governance forums • Strong troubleshooting and performance-tuning mindset Nice-to-Have Experience • MLflow and MLOps architectures • Real-time analytics and streaming pipelines • Multi-region or cross-account data architectures • Consulting or MSP delivery experience Originally posted on Himalayas