Lead Data Infrastructure Engineer

📋 Description

• • Own the end-to-end design, evolution, and operation of Stellic’s data platform, from ingestion through transformation, aggregation, and presentation to support internal decision-making and customer-facing insights.
• • Design, build, and maintain customer-facing data systems that extend the core data architecture directly to university partners, ensuring seamless integration and value delivery.
• • Build and maintain scalable, reliable data processing systems with clear guarantees around correctness, consistency, and behavior under increasing scale and complexity.
• • Design flexible yet standardized data models and pipelines capable of accommodating diverse regional, institutional, and partner-specific data requirements while maintaining system integrity.
• • Ensure all data systems comply with regional and partner-specific data governance, privacy, and compliance requirements, including handling varied regulatory constraints across 7 countries.
• • Lead platform evolution by identifying gaps, driving architectural improvements, and establishing patterns for data correctness, lineage, validation, and long-term maintainability.
• • Collaborate closely with product, design, and customer-facing teams to translate business and user needs into technical specifications for data products and infrastructure.
• • Implement and manage data pipelines using Python and SQL, with tools including dbt for transformations, MWAA/Airflow for orchestration, and Pydantic for schema validation.
• • Manage data storage and cataloging systems using S3, Redshift Serverless, Glue Data Catalog, and Schema Registry, ensuring efficient access and governance.
• • Operate within an AWS cloud environment utilizing Terraform for infrastructure-as-code, Lake Formation for ABAC-based governance, and GitHub for version control.
• • Monitor system health and performance using observability tools including Sentry, Grafana, Prometheus, PostHog, and Metabase.
• • Support both single-tenant and multi-tenant data architectures to serve diverse university partners with varying technical and operational needs.
• • Drive technical decision-making with a focus on pragmatic, high-velocity delivery while maintaining uncompromising quality and alignment with Stellic’s mission to improve student outcomes.
• • Mentor and guide junior engineers on best practices in data engineering, system design, and operational excellence within a startup environment.
• • Translate complex data requirements from academic institutions into technical solutions that balance flexibility with standardization without compromising compliance or scalability.
• • Continuously improve data lineage tracking and metadata management to enhance transparency and trust in data used for critical institutional and student-facing decisions.