EverCommerce - Senior Data Engineer

📋 Description

• • Design, build, and operate scalable batch and streaming data pipelines to support analytics, real-time insights, and AI-driven products across the EverCommerce ecosystem.
• • Lead architecture decisions for Lakehouse-based data platforms using Apache Iceberg and modern data storage patterns.
• • Develop and orchestrate complex data workflows using Apache Airflow to ensure reliability, scheduling, and error handling across distributed systems.
• • Build and maintain analytics-ready datasets through DBT for transformation, modeling, and semantic layer development to enable self-service analytics.
• • Develop and manage real-time data pipelines using Kafka to support low-latency ingestion and event-driven processing needs.
• • Utilize Databricks for large-scale data processing, advanced analytics, and enabling AI/GenAI use cases on the platform.
• • Ingest and manage structured and unstructured data from diverse sources using Fivetran and other managed data lake tools.
• • Design and optimize data storage solutions within Lakehouse architecture to improve performance, cost-efficiency, and scalability.
• • Implement comprehensive data quality frameworks, automated testing, and observability tools to ensure trusted, accurate, and reliable data outputs.
• • Optimize performance and reduce costs across AWS services including Athena, EC2, S3, and related infrastructure components.
• • Collaborate cross-functionally with BI, product, and engineering teams to deliver actionable data solutions aligned with business objectives.
• • Mentor junior data engineers and contribute to the establishment of engineering best practices, standards, and code reviews.
• • Drive improvements in developer productivity through automation of pipeline generation, QA workflows, documentation, and lineage tracking.
• • Maintain and enhance a semantic layer to unify data definitions and empower non-technical users with consistent, trusted reporting.
• • Ensure data governance, compliance, and secure access patterns are embedded into all data systems and pipelines.
• • Support the evolution of metadata-driven and declarative pipeline frameworks to increase platform agility and reduce manual intervention.
• • Enable AI-assisted development tools to accelerate ingestion-to-consumption workflows and automate DBT model generation and testing processes.
• • Work within a fully remote, US-based team structure with occasional travel up to four times per year for quarterly planning or corporate events.