Data Engineer

📋 Description

• • Own the end-to-end lifecycle of healthcare claims data at Public Interest Company—ingesting raw feeds from payers, clearinghouses, and providers and turning them into clean, standardized, analytics-ready assets that directly fuel our AI-driven recovery engine.
• • Architect, build, and continuously improve scalable ETL/ELT pipelines in Databricks using SQL, Python, and dbt; ensure every claim, member, provider, and diagnosis code lands in the right table at the right time with full lineage and auditability.
• • Implement automated data-quality gates that catch anomalies before they reach downstream legal, actuarial, and product teams; write unit tests, data-diff checks, and freshness monitors that alert the team within minutes—not days—when something drifts.
• • Partner with legal-tech analysts and clinical reviewers to translate complex business rules (COB, subrogation, lien statutes) into deterministic SQL logic, enabling the platform to flag high-value recovery opportunities worth millions in mis-paid claims.
• • Optimize pipeline performance by profiling Spark jobs, tuning file formats (Delta, Parquet), and leveraging Databricks’ auto-scaling clusters to cut runtimes and cost per claim processed by double-digit percentages.
• • Maintain comprehensive documentation and a living data-dictionary so that new hires, auditors, and client data-scientists can onboard in hours instead of weeks.
• • Serve as the first line of defense during data incidents: triage root causes, roll out hot-fixes, and run post-mortems that prevent recurrence while keeping stakeholders informed with clear, jargon-free updates.
• • Contribute to the evolution of our data architecture—evaluate new tooling (e.g., lakehouse features, streaming ingestion, data-contract frameworks) and run proof-of-concepts that keep the stack modern and maintainable.
• • Collaborate closely with product managers and client-success teams to scope new data sources (Medicare, Medicaid, commercial payers) and deliver them on aggressive startup timelines without compromising quality.
• • Champion a culture of reproducibility and continuous improvement: peer-review PRs, share lunch-and-learn sessions on dbt packages or Python testing libraries, and mentor junior analysts who want to level up into full-fledged data engineers.
• • Ensure compliance with HIPAA, SOC 2, and state privacy regulations by embedding encryption, access controls, and de-identification routines into every pipeline stage.
• • Translate technical wins into business impact—publish monthly dashboards that show stakeholders exactly how many dollars were recovered because of cleaner, faster, more complete data.