Senior Data Engineer

📋 Description

• • As a Senior Data Engineer at Veriff OÜ, you will play a pivotal role in shaping and advancing our analytical data platform, the very foundation that empowers machine learning, data science, analytics, and reporting across the entire organization. Your expertise will be instrumental in building and maintaining robust, scalable, and reliable data infrastructure, ensuring that Veriffians have seamless access to accurate and timely data to drive critical business decisions and innovations.
• • You will take ownership of our data lake and data warehouse infrastructure, driving forward platform-level data management, governance, and reliability at an enterprise scale. This involves not just maintaining existing systems but actively evolving them to meet the growing demands of a fast-paced, data-driven company.
• • A core responsibility will be the design and maintenance of sophisticated data transformation pipelines. You will leverage your proficiency in Python and SQL to build efficient and maintainable ETL/ELT processes, with a keen focus on critical aspects such as schema evolution, optimal partitioning strategies, data compaction, and effective deduplication techniques to ensure data integrity and performance.
• • You will be tasked with implementing and optimizing storage formats, including columnar formats like Parquet and ORC, as well as Avro. This includes developing and applying advanced partitioning strategies and indexing methods to significantly enhance query performance and, crucially, reduce overall platform costs. Your work will directly impact the efficiency and economic viability of our data operations.
• • A significant part of your role will involve driving forward data governance initiatives. This includes implementing robust mechanisms for PII (Personally Identifiable Information) detection and classification, establishing stringent access control policies, developing comprehensive data cataloging solutions, ensuring accurate lineage tracking, and building effective data quality frameworks. Your efforts will be key to maintaining compliance and trust in our data assets.
• • You will be the guardian of the data platform's availability, reliability, and cost efficiency. This entails establishing and maintaining comprehensive observability, monitoring, and alerting systems to ensure the health and performance of both data pipelines and query engines. Proactive identification and resolution of issues will be paramount.
• • Collaboration is central to this role. You will work closely with Machine Learning Engineers, Data Scientists, Analysts, Product Managers, and other Engineering teams. Your goal will be to define clear data contracts, maintain schema consistency across different data domains, and deliver clean, well-governed datasets that meet the specific needs of these internal stakeholders.
• • You will also contribute to the strategic development of our disaster recovery strategy and the multi-region reliability of the data platform, ensuring business continuity and resilience in the face of potential disruptions.
• • This role offers a unique opportunity to work with cutting-edge technologies such as Spark, Apache Iceberg, S3, Trino/Athena, and Redshift, pushing the boundaries of what's possible in large-scale data processing and analytics.
• • By joining Veriff, you will be contributing to a mission of protecting honest people online, a critical endeavor in today's digital landscape. Your work will directly support Veriff's position as a leading identity verification platform partner for innovative organizations worldwide.