Senior Data Engineer

BrightSpring Health Services

We are seeking an experienced Senior Data Engineer to join our team in building a next generation AI-enabled Health Cloud Platform. This platform powers intelligent clinical and pharmacy solutions using Generative AI, Agentic AI, Robotic Process Automation (RPA), and Retrieval-Augmented Generation (RAG) with domain-specific vector databases.

As a Senior Data Engineer, you will design and optimize data pipelines, data lakes, and analytics frameworks that enable scalable AI/ML workflows. You will work closely with AI architects, data scientists, and product teams to ensure high-quality, secure, and compliant data infrastructure for healthcare applications.

Design, build, and maintain data pipelines for ingesting, transforming, and storing large-scale healthcare datasets.
• Implement ETL/ELT workflows using Databricks, Spark, and Delta Lake for high performance data processing.
• Optimize data architecture for AI/ML workloads, including support for vector databases and RAG pipelines.
• Collaborate with AI teams to integrate structured and unstructured data into predictive and generative AI models.
• Ensure compliance with healthcare data standards (FHIR, HL7, RxNorm) and regulatory requirements (HIPAA).
• Develop and enforce data governance, security, and quality frameworks.
• Monitor and improve data pipeline performance, scalability, and reliability.
• Mentor junior engineers and contribute to best practices in data engineering and MLOps.

• Bachelor's or Master's degree in Computer Science, Data Engineering, or related field.
• 10+ years of experience in data engineering with strong expertise in Databricks, Spark, and distributed data systems.
• Proficiency in Python, SQL, and experience with data modeling and data lakehouse architectures.
• Hands-on experience with cloud platforms (Azure preferred; AWS/GCP acceptable).
• Experience with vector databases (FAISS, Weaviate, Pinecone) and RAG pipelines.
• Familiarity with MLOps tools (MLflow, Airflow, or similar) and data versioning.
• Strong understanding of healthcare data standards and compliance frameworks.
• Excellent problem-solving skills and ability to work in a fast-paced, mission-driven environment.
• Knowledge of Generative AI and Agentic AI frameworks (LangChain, AutoGen).
• Familiarity with data security, privacy, and regulatory compliance in healthcare.
• Contributions to open-source projects or experience in large-scale data environments.
• Travel up to 25%