AI Engineer, Quality (Evals)

📋 Description

• • As an AI Engineer, Quality (Evals) at Fieldguide, you will own the evaluation infrastructure that ensures AI agents perform reliably at enterprise scale, building unified platforms, automated pipelines, and production feedback loops to evaluate new models against critical workflows within hours.
• • You will design and build observability systems, integrate with LangSmith and LangGraph, translate customer problems into agent behaviors, and orchestrate LLMs, tools, retrieval systems, and logic into reliable agent experiences while working at the intersection of ML engineering, observability, and quality assurance.
• • Fieldguide is a remote-first, Series B-backed AI startup headquartered in San Francisco, building software for audit and assurance professionals in cybersecurity, privacy, and financial audit, trusted by over 50 of the top 100 accounting and consulting firms to power mission-critical work.
• • You will gain ownership of large product areas, define evaluation standards for the engineering organization, advocate for evaluation-driven development, and directly impact leadership’s ability to communicate AI quality to boards and customers through high-visibility, production-focused work.