Databricks Platform Engineer

Data Engineering Responsibilities

* Design and build batch data pipelines using Apache Spark on EKS and Databricks, processing data from NAS and upstream source systems into Iceberg tables on S3

* Own the Bronze Silver Gold DLT pipeline architecture for assigned business domains

* Write and maintain DDL for Iceberg tables following the CDP naming standard (catalog.schema.table with region, LOB, and layer components)

* Integrate with existing loading framework (Autosys OpenShift S3 Airflow DAG trigger chain) and extend it as the platform evolves

* Work with Data Stewards and DBA team to ensure schema design meets data classification requirements (INT, PII, CPII, CSPII)

* Implement and validate Unity Catalog governance controls - column masking, row-level filters, and lineage tracking - for tables under your ownership

* Contribute to the migration of pipeline artifacts from DBFS to UC Volumes as part of the platform's governance uplift programmer

DataOps Responsibilities

* Own the operational health of pipelines in your domain - monitoring, alerting, failure triage, and restart procedures

* Build and maintain job monitoring integration between Databricks / Airflow and the team's observability stack (Splunk, CloudWatch, MSK audit events)

* Define and document incident runbooks for common failure patterns: EKS resource exhaustion, S3 VPC Endpoint connectivity, Spark executor out-of-memory, DLT pipeline schema evolution failures

* Manage deployment of pipeline code and DAG updates through the Lightspeed CI/CD pipeline - understand what goes to S3 config bucket, what goes to the DAG bucket, and what goes to EFS

* Participate in on-call rotation for production pipeline incidents

* Work with the platform team on capacity planning for EKS cluster - there is no YARN queue here, resource exhaustion means job failure

* Maintain data quality monitoring and reconciliation between on-prem and cloud outputs during the migration period

What We Are Looking For

Must Have

* Overall 10 years' experience; 4+ years of data engineering experience, with at least 2 years working with Spark in a production environment

* Solid understanding of cloud networking fundamentals - VPC, subnets, VPC Endpoints, Security Groups. You do not need to be a network engineer, but you need to understand why a Spark pod's network path matters and what breaks when it is wrong

* Hands-on experience with AWS services: S3, EKS, IAM, KMS, SQS, MSK

* Experience with Apache Iceberg or Delta Lake in production - not just awareness, but actual table design and maintenance

* Familiarity with Kubernetes - you do not need to administer a cluster, but you need to understand pod l ifecycle, resource limits, and how to read a pod describe output when something fails

* Comfortable reading and writing Python and SQL at a production standard

* Understanding of IAM and credential management - why static keys are unacceptable, how STS AssumeRole works, what Vault does

Strong Preference

* Experience with Databricks, specifically Unity Catalog, MLflow, and DLT

* Experience working in a regulated financial services environment - you understand why data classification, audit trails, and access controls are not optional extras

* Familiarity with Airflow (DAG design, KubernetesPodOperator, trigger chain design)

* Experience with Spark performance tuning - you know what data skew looks like, when to add executors vs CPU, and how to read a Spark history server

* Exposure to model governance or MLOps - understanding how MLflow Model Registry, Champion/Challenger alias management, and MRM review processes fit together

Not Required But Noticed

* Experience with Lakebase or other OLTP-on-lakehouse technologies

* Familiarity with Collibra or similar enterprise data governance platforms

* Experience designing UC group naming conventions or RBAC frameworks at scale

What You Will Not Be Doing

* Building dashboards or BI reports

* Managing the Databricks platform itself - that sits with the platform team

The Environment

* Compute: EKS (Spark), OpenShift (on-prem loading framework), Databricks (UC, MLflow, DLT)

* Storage: S3 (Iceberg), EFS (intermediate artifacts), NAS (source files)

* Orchestration: Airflow (MWA), Autosys (on-prem, being phased out)

* Messaging: MSK (Kafka), SQS, SNS

* Catalog: HMS on Aurora migrating to AWS Glue Unity Catalog

* Security: Vault (credential injection), KMS (encryption), PrivateLink (no public internet for data paths), CloudTrail (audit)

* CI/CD: Lightspeed (Citi internal), Bitbucket / GitHub

What Good Looks Like in This Role

In the first 90 days, you should be able to:

* Explain the end-to-end data flow from NAS to an Iceberg table in S3, naming every component the data touches and the network path it takes

* Run a Spark job on EKS independently, debug a failure from the pod logs, and know when to look at the Spark History Server vs CloudTrail

* Identify a governance gap in a UC table configuration and propose a fix

In six months, you should own at least one business domain end-to-end - pipeline design, operational health, schema governance, and incident response.

Salary Range: $125,000 to $140,000 per year