Data Engineer

📋 Description

• • Architect and own the future-state data interoperability platform that powers critical public-health decisions across the United States. You will design end-to-end, secure, high-performance pipelines that move millions of sensitive health records daily—transforming raw data into actionable insights that directly influence how state, tribal, local, territorial, and federal agencies respond to outbreaks, chronic disease trends, and emerging threats.
• • Lead the modernization of legacy ingestion and transformation systems by re-platforming them onto cloud-native, event-driven architectures. You will map existing data flows, identify bottlenecks, and craft migration roadmaps that minimize downtime while maximizing scalability, reliability, and compliance with HIPAA, FISMA, and CDC security frameworks.
• • Build and operate both real-time and batch ETL/ELT pipelines using Apache NiFi, Kafka, Python, and SQL. You will tune partitioning strategies, optimize serialization formats, and implement exactly-once semantics to guarantee data integrity across disparate sources such as syndromic surveillance feeds, electronic lab reports, and immunization registries.
• • Establish rigorous data-quality and observability frameworks. You will define SLIs/SLOs, create automated anomaly detection, author runbooks, and stand up alerting channels (PagerDuty, Slack) so that on-call engineers can triage and resolve issues before they impact public-health analysts or downstream machine-learning models.
• • Collaborate daily with data architects, integration engineers, data scientists, and product managers in agile squads. You will translate complex epidemiological requirements into technical user stories, review pull requests, and mentor junior engineers on best practices for schema evolution, idempotent transformations, and secure credential management.
• • Champion security-by-design principles throughout the data lifecycle. You will implement field-level encryption, role-based access controls, and data-deidentification routines that satisfy both federal mandates and state-level privacy regulations, ensuring that every record is protected without hindering timely analytics.
• • Drive continuous improvement through experimentation and innovation. You will prototype new ingestion patterns (e.g., change-data-capture via Debezium), evaluate emerging storage formats (Iceberg, Delta), and present findings in internal tech talks that raise the technical bar across the entire engineering organization.
• • Contribute to containerized deployments and CI/CD workflows. While DevOps/SRE owns the platform layer, you will package pipeline components as Docker images, write Helm charts, and integrate automated tests into GitHub Actions so that every code change is validated, scanned, and deployed safely to production.
• • Influence product strategy with data-driven insights. By surfacing latency, throughput, and cost metrics on executive dashboards, you will help prioritize roadmap items that deliver the greatest impact to public-health outcomes, ensuring that engineering efforts remain tightly aligned with InductiveHealth’s mission of saving lives through better data.