Lead Data Engineer(Snowflake/DBT)

📋 Description

• • As a Lead Data Engineer at Tide Platform Limited, you will be instrumental in architecting, building, and maintaining the robust data pipelines and services that power our business intelligence, reporting, and analytical functions. You will play a pivotal role in shaping our data infrastructure, ensuring its scalability, reliability, and efficiency to support Tide's mission of helping SMEs save time and money.
• • You will be responsible for developing end-to-end ETL/ELT pipelines, collaborating closely with Data Analysts from various business functions to translate their needs into actionable data solutions. This involves designing, developing, and implementing automated processes for data extraction, transformation, and loading, with a strong emphasis on a Data Mesh architecture.
• • A key aspect of this role is to act as a technical leader and mentor for junior engineers within the team. You will be the go-to expert for data technologies and solutions, providing guidance, sharing best practices, and fostering a culture of continuous learning and improvement.
• • You will be expected to provide on-the-ground troubleshooting and diagnosis for architecture and design challenges, proactively identifying and resolving technical issues as they arise to ensure the smooth operation of our data systems.
• • Continuously seek opportunities to enhance both the 'what' and the 'how' of data pipeline delivery. This includes evaluating new technologies, optimizing existing processes, and implementing innovative solutions to improve data quality, performance, and accessibility.
• • Translate complex business requirements into detailed technical specifications. This involves defining data models, specifying dbt models to be built, setting up timings, implementing tests, and configuring reports to meet business needs accurately.
• • Take ownership of the end-to-end delivery of data models and reports, from initial conception and development through to deployment and ongoing maintenance.
• • Perform exploratory data analysis to identify potential data quality issues early in the pipeline. Implement robust testing strategies to prevent future occurrences and ensure the integrity of our data.
• • Work collaboratively with Data Analysts to ensure all data feeds are optimized for performance and are available at the required times. This includes implementing efficient data capture, change data control, and other delta loading approaches.
• • Manage the entire lifecycle of data sources, including discovery, transformation, testing, deployment, and comprehensive documentation.
• • Champion and help define data warehouse governance standards, focusing on data quality, rigorous testing, adherence to coding best practices, and conducting thorough peer reviews to maintain high standards across the team.
• • Develop Looker dashboards for specific use cases as required, ensuring they provide clear, actionable insights to business stakeholders.
• • Contribute to the strategic direction of Tide's data platform, leveraging your expertise to influence architectural decisions and technology choices.
• • You will be working with a modern data stack including Snowflake, dbt, Airflow, Fivetran, SQL, and Looker, with a cloud environment primarily on AWS and some GCP. This role offers a unique opportunity to work in a fast-paced, data-driven fintech environment, contributing directly to the success of small and medium-sized enterprises globally.
• • Embrace agile methodologies, such as Scrum, to ensure efficient and iterative development cycles. Participate actively in sprint planning, daily stand-ups, and retrospectives.
• • Maintain high development standards, with a strong focus on code quality, comprehensive unit testing, continuous integration, and deployment (CI/CD) practices.
• • Develop and maintain clear, precise technical documentation, effectively communicating complex technical concepts to both technical and non-technical stakeholders.
• • Collaborate with cross-functional teams, including product, engineering, and business intelligence, to ensure data solutions align with overall business objectives.
• • Contribute to the evolution of our Data Mesh architecture, ensuring data products are discoverable, addressable, trustworthy, and secure.
• • Proactively identify and address performance bottlenecks in data pipelines and queries, optimizing for speed and cost-efficiency.
• • Stay abreast of the latest trends and technologies in the data engineering space, bringing new ideas and approaches to the team.
• • Mentor and guide junior data engineers, fostering their technical growth and development within the team.
• • Ensure data privacy and security best practices are embedded in all data engineering processes and solutions.
• • Drive initiatives to improve data observability and monitoring, enabling proactive issue detection and resolution.
• • Contribute to the development and maintenance of data dictionaries and business glossaries to enhance data understanding and usability across the organization.
• • Participate in architectural discussions and contribute to the long-term vision for Tide's data infrastructure.
• • You will be a key player in ensuring that Tide's data infrastructure can scale effectively to support our rapid global growth and evolving business needs.