Software Engineering & DevOps AI Rater/Evaluator

📋 Description

• • Software Engineering & DevOps AI Rater/Evaluator role at Lilt Production involves evaluating AI outputs related to software engineering, DevOps, and infrastructure topics using structured rubrics to assess technical correctness, security implications, and best-practice alignment.
• • Day-to-day responsibilities include performing structured scoring, classification, and judgment tasks; identifying hallucinations, incorrect code, unsafe recommendations; validating and refining evaluation rubrics; conducting error analysis; and partnering with research and product teams on evaluation design depending on track (Rater or Senior Evaluator).
• • Lilt Production is building a global network of domain experts to support high-quality AI evaluation for enterprises and hyperscalers, ensuring multilingual AI systems are accurate, safe, and aligned with real-world engineering practices.
• • This role offers the opportunity to directly influence multilingual AI model quality, safety, and deployment readiness while working flexibly on contract-based, project-driven tasks with clear expectations and timely compensation.