Observability Infrastructure Engineer

📋 Description

• • Adyen is a leading financial technology platform providing integrated payments, data, and financial products to global giants like Facebook, Uber, H&M, and Microsoft. We engineer everything for ambition, fostering an environment where our people can thrive, own their careers, and tackle unique technical challenges at scale as a team. We deliver innovative and ethical solutions that empower businesses to achieve their goals faster.
• • We are seeking an experienced Observability Infrastructure Engineer to join our dynamic Platform Engineering organization. This role is pivotal in building and managing the core Observability pillars, both on-premise and within our Kubernetes environments. Our systems are the lifeblood of hundreds of product teams, collecting, processing, and storing the logs, metrics, and traces essential for real-time service monitoring.
• • This is an exceptional opportunity for a builder and a dedicated problem solver who thrives on deep technical troubleshooting across complex distributed systems. You will be instrumental in transforming recurring issues into automated, repeatable solutions, operating within a large-scale environment that manages petabytes of data and thousands of servers. We are currently undergoing a significant transformation, with a strong focus on automating operations and enabling self-service capabilities for our users.
• • **Build the Next Generation of Our Platform:** You will play a crucial role in designing and implementing the future architecture of our logging and metrics systems. This includes architecting solutions to support new global regions, ensuring robust data isolation, and maintaining strict regulatory compliance across diverse geographies. Your contributions will shape the scalability and resilience of our observability infrastructure for years to come.
• • **Own Infrastructure Operations:** Take full ownership of our hybrid infrastructure, encompassing the complete lifecycle management of over 1,500 servers. This includes managing both bare-metal and Kubernetes environments, ensuring their stability, performance, and security.
• • **Automate to Reduce Toil:** Leverage your coding expertise in Go or Python to eliminate manual operational tasks. Your primary objective will be to engineer self-healing systems that minimize or eliminate the need for manual intervention, especially during off-hours. You will also enhance our CI/CD pipelines to ensure that changes to our clusters are safe, predictable, and fully automated, streamlining deployments and reducing operational overhead.
• • **Optimize for Scale and Performance:** Dive deep into identifying and resolving performance bottlenecks within our distributed tracing and logging pipelines. You will tackle challenges posed by high-volume data streams that can overwhelm standard configurations. This involves tuning our Elasticsearch clusters, optimizing Prometheus and VictoriaMetrics storage solutions, and ensuring our OpenTelemetry implementation can seamlessly handle peak traffic without data loss or performance degradation.
• • **Reliability and Engineering Focus:** While participating in on-call rotations is part of the role, your main focus will be on engineering robust solutions that proactively prevent alerts from firing. You will lead efforts to upgrade our technology stack to the latest versions, ensuring our platform remains secure, performant, and up-to-date. Enhance the self-service experience for our users by implementing automated guardrails and effective quota management systems. This will prevent noisy tenants from impacting platform stability and allow for the design of safer API access patterns, empowering users while maintaining system integrity.
• • **Collaborate and Innovate:** Work closely with product teams to understand their observability needs and translate them into scalable infrastructure solutions. Contribute to architectural discussions, share knowledge, and mentor junior engineers. Embrace a culture of continuous improvement and innovation, always seeking better ways to manage and enhance our observability platform.