AI Evaluation Engineer - Proofline

Fermi AI, Inc.

Job Overview

Location

Bangalore, India

Job Type

Full-time

Full Job Description

📋 Description

• As an AI Evaluation Engineer, you will take ownership of the evaluation scaffolding and quality layer, including eval harnesses for AI-facing features, test infrastructure, and release gates.
• You will play a critical role in monitoring how our product behaves across AI assistants (e.g., Claude, ChatGPT), accounting for differences by host and continual changes.
• You will build and maintain evaluation harnesses for AI-facing features to measure and tune system quality (e.g., capture quality, retrieval quality, guidance quality).
• You will own end-to-end and API test infrastructure (Playwright-class), supporting a continuous, daily-release cycle.
• You will design and execute host-behavior probes using scripted user sessions across diverse AI assistants, ensuring product behavior aligns with expectations.
• You will gate production releases through thorough user acceptance testing (UAT), regression analysis, and quality reporting.

🎯 Requirements

• Background in SDET/QA automation or ML evaluation, with proven ownership of test or evaluation infrastructure and not just executing tests.
• Strong programming skills in Python or TypeScript, with experience in API-level testing and tools like Playwright or Cypress.
• Familiarity with LLM applications or a demonstrated interest in evaluating non-deterministic systems.
• Highly autonomous and able to define your own workflows and processes for evaluation and verification.

🏖️ Benefits

• This role is onsite in Bangalore, India.
• You will be working in a small team with high trust and written decisions.
• Designs get adversarial review before code; PRs get automated review driven to zero open findings; features aren't done until verified on a live system.
• AI agents do a large share of the implementation—your leverage is judgment: framing the problem, freezing the right design, and knowing when the machine is wrong.

Skills & Technologies

Python

TypeScript

Cypress

Remote

Ready to Apply?

Apply Externally

You will be redirected to an external site to apply.

AI Job Fit Analysis

Pro

See exactly how your profile matches this role — strengths, skill gaps, and what to do about them.

Fermi AI, Inc.

Visit Website

About Fermi AI, Inc.

Fermi AI is a company focused on developing advanced artificial intelligence solutions. They specialize in creating AI models and platforms designed to accelerate scientific discovery and innovation across various fields. Their technology aims to empower researchers and developers by providing powerful tools for data analysis, simulation, and prediction. Fermi AI's mission is to leverage the transformative potential of AI to tackle complex challenges and drive progress in areas such as materials science, drug discovery, and climate modeling. The company is committed to building robust, scalable, and interpretable AI systems that can unlock new insights and accelerate the pace of scientific breakthroughs.

View Company Profile

Get more remote jobs like this

Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.

Weekly remote jobs and featured talent.

No spam. Only curated remote roles and product updates. You can unsubscribe anytime.