Data Engineer

📋 Description

• • Architect and deliver end-to-end data solutions that power high-impact analytics and AI initiatives for a federal public-facing agency. You will own the full lifecycle—from ingestion through transformation, storage, and governance—ensuring data is accurate, secure, and immediately actionable.
• • Partner daily with AI & ML engineers to operationalize Python-native machine-learning pipelines. Translate research notebooks into production-grade workflows, optimize feature-engineering code, and guarantee reproducibility, scalability, and compliance with federal security standards.
• • Design and maintain scalable ETL/ELT pipelines using Databricks and AWS services (S3, Lambda, Glue, EMR). Leverage Delta Live Tables, DBT, and Spark Structured Streaming to process both batch and real-time data with sub-second latency where required.
• • Implement change-data-capture (CDC) patterns and efficient ingestion strategies that keep downstream analytics in sync with source systems. Automate schema evolution, data-quality checks, and anomaly detection to reduce manual intervention and accelerate insight delivery.
• • Write, refactor, and harden Python data-processing scripts for AWS Lambda, ensuring idempotency, fault tolerance, and cost efficiency. Enforce rigorous unit-test coverage for all Spark, Python, and Lambda code to guarantee reliability in mission-critical environments.
• • Establish and enforce data-lifecycle policies—retention, archival, backup, restore—aligned with federal mandates and agency security frameworks. Build self-service tooling that allows data owners to manage their own datasets while maintaining centralized governance.
• • Provide architectural leadership and hands-on technical support for integrating identity-management and security technologies. Serve as a subject-matter expert to senior executives, translating complex data concepts into clear, actionable recommendations.
• • Continuously identify bottlenecks in data pipelines and analytics workflows. Introduce automation, caching, and performance tuning that reduce runtimes and cloud spend while increasing throughput and reliability.
• • Communicate findings and progress through concise dashboards, executive briefings, and technical documentation. Tailor messaging to diverse audiences—from data scientists to C-suite stakeholders—ensuring alignment and transparency.
• • Champion a culture of continuous improvement and “good agile.” Mentor junior engineers, lead code reviews, and contribute to internal guilds that elevate engineering standards across Packaged Agile and client teams.