Data Engineer

📋 Description

• • Architect and maintain scalable data pipelines that ingest, cleanse, and transform multi-terabyte datasets from 30+ sources (EMRs, CRMs, prescription feeds, digital campaigns) into a unified, HIPAA-compliant data lake on AWS (S3, Glue, Redshift, Lambda).
• • Own the end-to-end reliability of critical data products that power real-time physician targeting, campaign attribution, and ROI dashboards used by 200+ pharma brands; ensure 99.9 % uptime and <5-minute SLA on freshness.
• • Write elegant, peer-reviewed SQL and Python/PySpark code that implements slowly-changing dimensions, idempotent ETL jobs, and automated data-quality monitors; version everything in Git and deploy via CI/CD (GitHub Actions, Terraform).
• • Partner with Analytics, Product, and Data-Science teams to translate vague business questions (“Which HCPs increased Rx after an email?”) into precise technical specs, then deliver tested, documented datasets they can trust.
• • Design and enforce role-based access controls (RBAC) across dev, staging, and prod environments; automate onboarding/off-boarding scripts that provision least-privilege Redshift schemas, IAM roles, and Snowflake warehouses in minutes, not hours.
• • Champion software craftsmanship: unit-test every transformation, enforce style guides, perform code reviews, and mentor junior engineers on clean architecture, functional programming, and the Zen of Python.
• • Continuously optimize cost and performance—right-size Redshift clusters, implement workload-management queues, partition Athena tables, and compress Parquet files—to keep cloud spend under $50k/month while doubling data volume.
• • Build self-healing data workflows using Airflow/Dagster: retry failed tasks, send Slack alerts with context, and maintain detailed lineage so stakeholders know exactly where their numbers come from.
• • Evaluate emerging tech (dbt, Lakehouse, Delta Lake, Flink) through quick POCs; present findings to the SVP and influence the 18-month technical roadmap.
• • Contribute to a remote-first culture: daily virtual stand-ups, fortnightly “data guild” demos, blameless post-mortems, and a weekly “learning stipend” hour to level up on GCP, Scala, or whatever sparks curiosity.