AI Automation Engineer _ Virtual

📋 Description

• • As an AI Automation Engineer within Alight's AI Center of Excellence (CoE), you will play a pivotal role in establishing and scaling the foundational automation framework for AI Quality. This is a unique opportunity to shape the future of AI quality assurance at Alight, working at the forefront of artificial intelligence technologies.
• • You will collaborate closely with the AI Quality Lead and a team of talented developers to design, implement, and maintain robust automated test suites specifically tailored for AI models and agents. Your contributions will be critical in ensuring the reliability, safety, and performance of our AI-driven solutions.
• • A core responsibility will be the development and ongoing maintenance of sophisticated automated test frameworks. These frameworks will support a variety of AI systems, including Large Language Models (LLMs), Retrieval Augmented Generation (RAG) systems, agentic AI, and traditional Machine Learning (ML) models. This involves understanding the unique testing challenges posed by each type of AI system.
• • You will be tasked with implementing a comprehensive suite of test cases, encompassing functional correctness, safety protocols, bias detection, prompt injection vulnerabilities, refusal correctness, and the accurate utilization of tools by AI agents. This requires a deep dive into the nuances of AI behavior and potential failure modes.
• • A key aspect of the role involves integrating cutting-edge evaluation tools such as promptfoo and Ragas into our Continuous Integration/Continuous Deployment (CI/CD) pipelines. This integration will enable efficient and automated quality assessments, allowing for rapid feedback loops and continuous improvement.
• • You will also be responsible for supporting batch quality assessments, ensuring that large-scale evaluations of AI models are conducted systematically and effectively.
• • Authoring and validating OpenAPI/JSON Schema contracts for model inputs and outputs is another crucial duty. You will enforce these schema checks within the pipelines to guarantee data integrity and predictable model interactions.
• • To enhance system monitoring and performance analysis, you will instrument observability metrics, including latency, success rates, operational costs, and data drift. Utilizing OpenTelemetry, you will export these metrics to dashboards, providing real-time insights into AI system performance.
• • You will actively support data quality assurance processes, including lineage tracking and drift monitoring, leveraging tools like Evidently and OpenLineage. This ensures the integrity and reliability of the data powering our AI models.
• • Collaboration is central to this role. You will work hand-in-hand with AI developers, Software Development Engineers in Test (SDETs), and Site Reliability Engineers (SREs) to troubleshoot issues, optimize automation processes, and meticulously document all automation procedures.
• • Furthermore, you will contribute to the development of red-team playbooks and security testbooks, aligning them with established risk frameworks such as OWASP and ATLAS. This proactive approach to security testing will help identify and mitigate potential vulnerabilities in our AI systems.
• • This role offers a significant opportunity to influence the direction of AI quality and automation within a growing organization, contributing to the development of cutting-edge AI solutions that impact millions of lives.
• • You will be at the forefront of ensuring responsible AI development, focusing on bias mitigation, fairness, and adherence to regulatory compliance standards.
• • The virtual nature of this role allows for flexible working arrangements, supporting a healthy work-life balance. Alight has a proven track record as a leader in flexible and remote work environments, recognized for its commitment to employee well-being.
• • By joining Alight, you become part of a company that champions people, encourages growth with purpose, and embodies integrity. You will contribute to a mission of unifying the benefits ecosystem and helping clients build a healthy and financially secure workforce.