AI Evaluation Engineer (Software Engineering / Code)

📋 Description

• • AI Evaluation Engineer specialized in software engineering to design benchmark tasks based on real-world coding workflows, ensuring AI systems can analyze, modify, and validate code changes accurately.
• • Design and build multi-agent benchmark tasks using real-world code changes (bug fixes, migrations, refactors); work with the Harbor evaluation framework to run and validate tasks in containerized environments; write clear technical specifications and Python-based verification scripts; define task decomposition strategies across multiple agents; analyze large open-source codebases to extract realistic scenarios; run, debug, and refine tasks in Docker environments for reproducibility; improve task quality based on evaluation results.
• • Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions, helping companies build high-performing teams by matching them with professionals who truly fit their needs, with a strong background in software engineering and leadership.
• • Develop expertise in AI evaluation frameworks, multi-agent systems, and large-scale code analysis; gain hands-on experience with Docker, Harbor, and benchmark design for LLMs; contribute to advancing AI coding evaluation standards while working remotely with a global team across multiple time zones.