Lead Data Engineer

📋 Description

• • Join AirOps, the pioneering end-to-end content engineering platform designed for the AI era, and play a pivotal role in shaping the future of how brands get found and stay found in an AI-driven discovery landscape.
• • As the Lead Data Engineer, you will be entrusted with the ownership and scaling of the critical data platform that underpins AirOps's sophisticated insights into AI search visibility and content performance.
• • This role offers a unique opportunity to set the technical direction for our data infrastructure, contribute directly through writing production-level code, and build a high-performing, lean team dedicated to transforming raw web, content, and AI agent data into exceptionally trustworthy and actionable datasets.
• • Your contributions will directly fuel customer-facing analytics and innovative product features, while simultaneously providing our content and growth teams with a clear, data-driven pathway from strategic planning to flawless execution.
• • We are seeking an individual who embodies extreme ownership, possesses an unwavering commitment to data quality down to the finest details, and thrives on cross-functional collaboration to deliver results rapidly without compromising rigor.
• • **Data Platform Ownership:** You will be responsible for the end-to-end design, development, and operation of robust batch and streaming data pipelines. This includes ingesting diverse data sources such as web crawlers, third-party partner APIs, product analytics events, and CRM data, ensuring a comprehensive and reliable data foundation.
• • **Core Data Modeling:** Define, implement, and meticulously maintain company-wide data models that are crucial for understanding our core business. This encompasses entities like content, search queries, ranking positions, AI agent responses, user engagement metrics, and revenue attribution, providing a unified view of performance.
• • **Orchestration and CI/CD:** Implement and manage sophisticated workflow orchestration using tools like Airflow or Prefect. Drive data transformations using dbt, ensuring code is version-controlled, rigorously tested, and deployed through automated CI/CD processes for maximum efficiency and reliability.
• • **Data Quality and Observability:** Establish stringent Service Level Agreements (SLAs) for data freshness and accuracy. Implement comprehensive data quality tests and data contracts. Proactively monitor data lineage and freshness, and lead rapid root cause analysis for any data discrepancies or issues.
• • **Warehouse and Storage Management:** Oversee and optimize our data warehouse solutions, primarily Snowflake or BigQuery, alongside PostgreSQL for transactional use cases. Focus on achieving strong performance, efficient cost management, and implementing effective partitioning strategies to handle large volumes of data.
• • **Semantic Layer and Metrics Definition:** Deliver clear, well-documented, and easily accessible metrics datasets. This semantic layer will serve as the single source of truth for dashboards, A/B testing frameworks, and product activation metrics, empowering data-driven decision-making across the organization.
• • **Product and Customer Impact:** Collaborate closely with Product Management and Customer Success teams to define precise tracking plans for user behavior and content performance. Measure and report on the impact of content across both on-site and off-site channels, providing actionable insights for optimization.
• • **Tooling and Vendor Evaluation:** Evaluate, select, and integrate best-in-class tools for data ingestion, enrichment, observability, and reverse ETL, ensuring our technology stack remains cutting-edge and efficient.
• • **Team Leadership and Mentorship:** Lead the hiring, mentoring, and professional development of a small but impactful team of data and analytics engineers. Establish and enforce high standards for code quality, implement effective code review practices, and develop robust operational runbooks.
• • **Strategic Technical Direction:** Contribute to the overall data strategy of the company, influencing architectural decisions and technology choices to support hyper-growth and evolving business needs.
• • **Innovation and Best Practices:** Stay abreast of the latest trends and technologies in data engineering, AI, and content analytics, championing the adoption of best practices and innovative solutions within the team and across the organization.
• • **Cross-functional Collaboration:** Foster strong working relationships with engineering, product, marketing, and sales teams to ensure data initiatives are aligned with business objectives and deliver maximum value.
• • **Scalability and Performance:** Design and implement solutions that are inherently scalable and performant, anticipating future growth in data volume and complexity.
• • **Data Governance and Security:** Ensure that data is handled securely, ethically, and in compliance with relevant regulations, establishing clear data governance policies.
• • **Empowerment through Data:** Ultimately, your work will empower our customers and internal teams by providing them with the data and insights needed to succeed in the AI-driven content landscape.