AI Evaluation Engineer (Agentic Coding / Software Engineering)

📋 Description

• • AI Evaluation Engineer (Agentic Coding / Software Engineering) at Gramian Consultancy, a boutique firm specializing in IT professional services and engineering talent solutions, focused on evaluating and improving datasets for agentic coding models to enhance AI-driven software development.
• • Day-to-day responsibilities include executing coding tasks in agentic coding environments, reviewing and validating model-generated code trajectories, running tests and scripts to assess correctness, writing evidence-based evaluation rationales, designing realistic multi-step coding tasks, creating and refining evaluation rubrics, ensuring consistency and quality across assessments, and identifying and reporting issues in workflows or environments with clear evidence.
• • The team operates as a global contractor network supporting clients in building high-performing engineering teams, with this role contributing directly to the reliability and performance of AI coding tools through rigorous human-in-the-loop evaluation.
• • In this role, you will deepen your expertise in AI-assisted software engineering, gain hands-on experience evaluating cutting-edge agentic coding systems, refine your ability to assess code quality and correctness at scale, and contribute to shaping the future of AI-driven development workflows.