AI Quality Lead -Virtual

📋 Description

• • As the AI Quality Lead for the Artificial Intelligence Center of Excellence (AI CoE), you will be instrumental in architecting and driving the end-to-end AI Quality program from its inception. This is a pivotal, hands-on leadership role that demands both a strategic vision for establishing robust AI quality practices and the technical depth required to take Alight's AI Quality initiatives from concept to full implementation (0→1).
• • Your primary responsibility will be to design and implement a comprehensive, unified AI Quality strategy and Standard Operating Procedure (SOP). This strategy will encompass scaling quality controls effectively across different risk tiers, from Tier 1 (highest risk) to Tier 3 (lowest risk), ensuring that AI solutions meet rigorous standards.
• • You will define and own critical success metrics, establish clear risk tiering methodologies, set acceptance criteria for AI solutions, and define operating thresholds that guarantee performance, reliability, and safety.
• • A key aspect of this role involves leading the development of sophisticated test suites. This includes creating golden/adversarial test suites designed to uncover vulnerabilities and ensure robustness, covering areas such as prompt injection, misuse scenarios, safety protocols, bias detection, and functional automation.
• • You will oversee the creation and ongoing maintenance of essential testing components, including schema-level UI/API tests, agent contracts that define interactions, and comprehensive observability instrumentation to monitor AI system performance in real-time.
• • You will take ownership of the organization-wide evaluation harnesses, leveraging industry-standard tools such as promptfoo, TruLens, and Ragas. This includes managing CI gating processes to ensure quality is maintained throughout the development lifecycle and developing red-team playbooks to proactively identify and mitigate risks.
• • The role requires managing independent spot checks, ensuring the creation of detailed factsheets, maintaining registry entries for AI models and solutions, and generating evidence required for governance and audit purposes, thereby ensuring compliance and accountability.
• • You will foster strong collaborative relationships with Product Owners, Developers, Site Reliability Engineers (SRE)/Observability teams, and Governance stakeholders. This collaboration is crucial for ensuring compliance with internal policies and external regulations, and for driving operational excellence across all AI initiatives.
• • A significant part of your contribution will be driving continuous improvement in key AI quality dimensions, including fairness, explainability, safety, security, and operational Service Level Objectives (SLOs), ensuring that Alight's AI systems are not only functional but also ethical and trustworthy.
• • You will play a vital role in mentoring and growing the AI Quality team. This includes onboarding new team members, providing coaching and technical guidance to junior engineers, and fostering a culture of continuous learning and high performance within the team.
• • This position requires a deep understanding of various AI systems, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, agentic AI, or classic Machine Learning (ML) systems, enabling you to effectively lead quality initiatives across diverse AI applications.
• • You will be expected to be familiar with test automation frameworks, Continuous Integration/Continuous Deployment (CI/CD) pipelines, and various evaluation tools essential for AI quality assurance, such as promptfoo, TruLens, and Ragas.
• • A strong knowledge of fairness, explainability, safety, and security standards is paramount. This includes familiarity with established frameworks like OWASP Top 10, MITRE ATLAS, Fairlearn, and AIF360, ensuring AI systems are developed responsibly.
• • Experience with data integrity, data lineage tracking, and observability practices is essential. Familiarity with tools like OpenTelemetry, New Relic, and OpenLineage will be leveraged to ensure data quality and system transparency.
• • You should possess a working familiarity with AI infrastructure, orchestration tools, data integrity management, deployment strategies, and model registry practices, providing a holistic view of the AI lifecycle.
• • Knowledge of international standards and regulatory frameworks pertaining to Responsible AI is highly desirable, ensuring Alight remains at the forefront of ethical AI development and deployment.
• • The role also involves ensuring that AI solutions comply with relevant international standards and regulatory frameworks for Responsible AI, positioning Alight as a leader in ethical AI practices.
• • You will be responsible for ensuring that all AI solutions adhere to the highest standards of quality, reliability, and ethical considerations, contributing directly to Alight's reputation and success in the AI domain.