Senior Platform Engineer

Senior Platform Engineer

We are seeking a Senior Platform Engineer to build, administer, automate, secure, and operate enterprise Databricks and cloud data-platform environments. The role is responsible for Databricks workspace administration, Unity Catalog governance, identity and access management, compute and cluster policies, infrastructure automation, CI/CD enablement, monitoring, reliability, cost management, and production support.

The ideal candidate combines hands-on Databricks administration with strong AWS or Azure platform engineering, Terraform, Python, Kubernetes, CI/CD, security, networking, observability, and Site Reliability Engineering practices.

Key Responsibilities

Databricks Administration and Platform Engineering

* Administer Databricks accounts, workspaces, metastores, catalogs, schemas, external locations, storage credentials, connections, shares, recipients, and platform configurations.

* Provision and manage development, test, staging, and production workspaces using standardized, repeatable patterns.

* Configure workspace settings, repositories, jobs, notebooks, SQL warehouses, instance pools, job clusters, all-purpose compute, and serverless capabilities.

* Define and enforce cluster policies, approved runtime versions, libraries, init scripts, autoscaling, tagging, and compute-usage standards.

* Manage Databricks Runtime and platform upgrades, compatibility testing, release planning, maintenance windows, and rollback procedures.

* Support Databricks Workflows, Delta Live Tables or Lakeflow Declarative Pipelines, Databricks SQL, MLflow, model registry, feature engineering, and structured-streaming services.

* Troubleshoot workspace, permissions, connectivity, compute, storage, job execution, library, runtime, and performance issues.

* Maintain administration standards, platform documentation, knowledge articles, support procedures, and operational runbooks.

Unity Catalog, Identity, Security, and Governance

* Design and manage Unity Catalog metastores, catalogs, schemas, managed and external tables, volumes, external locations, storage credentials, and grants.

* Implement user and group provisioning through single sign-on, SCIM, identity-provider integration, and enterprise directory services.

* Administer account-level and workspace-level users, groups, service principals, permissions, entitlements, and access-control models.

* Implement role-based and attribute-based access controls, least-privilege permissions, separation of duties, and privileged-access procedures.

* Configure secure access patterns for secrets, tokens, credentials, service principals, private endpoints, storage, and external systems.

* Enable audit logging, lineage, system tables, tagging, data classification, row-level security, column masking, and compliance reporting.

* Partner with security and governance teams on encrypti on, key management, network controls, data loss prevention, retention, auditability, and regulatory requirements.

* Review access, monitor privileged activities, remediate policy violations, and support internal and external audits.

Cloud Infrastructure and Networking

* Build and operate Databricks on AWS or Azure, including secure integration with cloud storage, identity, networking, encryption, and monitoring services.

* On AWS, work with S3, IAM, KMS, VPC, PrivateLink, security groups, Route 53, CloudWatch, Secrets Manager, and related services.

* On Azure, work with ADLS Gen2, Microsoft Entra ID, managed identities, Key Vault, virtual networks, private endpoints, network security groups, Azure Monitor, and related services.

* Configure control-plane and data-plane connectivity, private networking, DNS, routing, firewall, proxy, and egress controls.

* Integrate Databricks with cloud data lakes, APIs, databases, message platforms, and enterprise applications.

* Support Kubernetes, Docker, EKS or AKS, and container-based platform services where required.

* Contribute to capacity planning, disaster recovery, high availability, backup, restoration, and business-continuity exercises.

Infrastructure as Code and Automation

* Build and maintain reusable Terraform modules using cloud and Databricks providers.

* Automate workspace, network, Unity Catalog, storage, identity, compute-policy, cluster, job, permission, and monitoring configurations.

* Manage Terraform state, workspaces, variables, modules, versioning, policy checks, drift detection, and controlled promotion across environments.

* Use Python, shell scripting, Databricks CLI, REST APIs, and SDKs to automate administrative and operational tasks.

* Implement self-service workspace, catalog, schema, access, and compute vending with appropriate approval and governance controls.

* Maintain configuration standards and reduce manual administration through repeatable automation.

DevOps, CI/CD, and Release Engineering

* Design and support CI/CD pipelines using GitHub Actions, GitLab CI/CD, Jenkins, Azure DevOps, or Harness.

* Automate deployment of notebooks, jobs, workflows, libraries, policies, infrastructure, and platform configuration.

* Support Databricks Asset Bundles, Git integration, artifact management, environment promotion, testing, approvals, and rollback.

* Integrate security scanning, policy validation, infrastructure testing, and release evidence into delivery pipelines.

* Enable engineering teams through templates, reusable pipelines, documentation, and self-service platform capabilities.

* Partner with application, data-engineering, and DevOps teams to ensure deploy ment standards are consistent and supportable.

Reliability, Monitoring, Operations, and FinOps

* Establish monitoring, alerting, dashboards, logs, metrics, traces, and health checks for Databricks and connected cloud services.

* Use CloudWatch, Azure Monitor, Datadog, Splunk, New Relic, or similar platforms to monitor availability, compute utilization, failures, security events, and cost.

* Define platform service-level indicators, service-level objectives, operational metrics, and error budgets.

* Lead incident response, problem management, root-cause analysis, corrective actions, and post-incident reviews.

* Manage vulnerability remediation, runtime patching, dependency updates, security exceptions, and platform lifecycle activities.

* Optimize cluster sizing, autoscaling, pools, SQL warehouses, job concurrency, serverless usage, storage, and workload scheduling.

* Implement budget controls, chargeback or showback tagging, utilization reporting, anomaly detection, and cost-optimization recommendations.

* Participate in operational support rotations and maintain escalation paths with Databricks and cloud providers.

* Coordinate platform upgrades, disaster-recovery tests, security reviews, and production-readiness assessments.

Collaboration and Technical Leadership

* Partner with architecture, data engineering, security, cloud, network, governance, FinOps, and service-management teams.

* Advise engineering teams on Databricks platform standards, secure patterns, deployment models, performance, and cost.

* Conduct technical reviews and ensure solutions meet enterprise architecture and operational-support requirements.

* Mentor platform engineers and administrators and lead knowledge-transfer sessions.

* Communicate platform health, risks, dependencies, incidents, and improvement roadmaps to technical and business stakeholders.

* Drive continuous improvement in automation, reliability, security, developer experience, and operational efficiency.

Required Qualifications

* Typically 7-10 years of cloud, DevOps, Site Reliability Engineering, infrastructure, or platform-engineering experience.

* At least 3 years of hands-on Databricks platform administration in an enterprise environment.

* Strong experience administering Databricks workspaces, Unity Catalog, compute, cluster policies, jobs, SQL warehouses, permissions, and service principals.

* Strong experience with AWS or Azure infrastructure, identity, storage, networking, encryption, monitoring, and private connectivity.

* Strong proficiency with Terraform and infrastructure-as-code practices.

* Experience with Python, shell scrip