Senior Data Engineer - Python and Terraform skills

📋 Description

• • As a Senior Data Engineer at Shift Technology SAS, you will be instrumental in shaping the future of our AI platform for insurance. You will play a pivotal role in designing, building, and maintaining the robust, scalable data infrastructure that powers our cutting-edge solutions in underwriting, claims, fraud detection, and risk management.
• • This position is at the heart of our engineering efforts, where complex data challenges are met with innovative solutions. You will collaborate closely with cross-functional teams, including Analytics, Machine Learning, Product, and Platform Engineering, to ensure our data systems not only meet but exceed the demands of our global clientele.
• • Your primary responsibility will be to drive the end-to-end data architecture design, ensuring systems are scalable, resilient, and performant. This involves architecting solutions that can handle vast amounts of data efficiently and reliably.
• • You will be tasked with designing, implementing, and maintaining sophisticated ETL/ELT pipelines. This includes developing new pipelines from scratch and optimizing existing ones to enhance efficiency and reduce operational costs.
• • A significant part of your role will involve owning and optimizing Apache Spark-based pipelines. This requires deep expertise in debugging, hardening, tuning, and performance optimization to ensure these critical data flows operate at peak efficiency.
• • You will translate complex business requirements into well-defined, scalable data models and warehouse schemas. This ensures that the data infrastructure is aligned with business objectives and can support advanced analytics and AI initiatives.
• • A key objective is to continuously improve pipeline performance, reliability, scalability, and cost-effectiveness. You will proactively identify bottlenecks and implement solutions to optimize resource utilization and operational expenditure.
• • You will lead or actively contribute to Proofs of Concept (POCs) and initiatives for adopting new technologies. This involves evaluating emerging tools and methodologies to enhance our data platform and maintain a competitive edge.
• • Ensuring best practices in data governance, data quality, observability, and monitoring is paramount. You will establish and enforce standards to maintain the integrity, trustworthiness, and operational visibility of our data assets.
• • You will work collaboratively with other engineering teams, product managers, and data scientists to integrate data solutions seamlessly into the broader platform and product offerings.
• • Mentoring and supporting junior engineers is an integral part of this senior role. You will provide technical guidance, conduct code reviews, and foster a culture of continuous learning and knowledge sharing within the team.
• • Taking ownership of infrastructure provisioning is crucial. You will be empowered to unblock yourself and your team by managing the necessary infrastructure resources, leveraging Infrastructure as Code principles.
• • You will be responsible for actively documenting and diagramming the technical design and architecture of our data systems. Clear and comprehensive documentation is essential for maintainability, knowledge transfer, and future development.
• • This role offers a unique opportunity to work with a modern tech stack including .NET Core (C#), Python, MS SQL Server, Spark, Azure Data Factory, Databricks, GoodData, and Terraform, with a strong focus on distributed data systems on cloud platforms, encompassing both batch and real-time processing challenges.
• • You will contribute to building powerful, scalable software and data platforms that are at the core of the value Shift Technology provides to its customers, reshaping how data drives decision-making in the insurance industry.