Job Overview
Location
Palo Alto
Job Type
Full-time
Category
Data Science
Date Posted
August 11, 2026
Full Job Description
📋 Description
- • We're a ~25-person AI startup building autonomous agents that handle real, complex work — email, calendar, browser automation, business software, and more.
- • Your mission: measure, understand, and continuously improve the quality of our AI agent capabilities.
- • You'll turn ambiguous product behavior into rigorous, measurable definitions of success — then build the evaluation infrastructure, datasets, metrics, and feedback loops that drive engineering and product decisions.
- • This is applied data science at the intersection of LLM systems, evaluation design, and production quality engineering.
- • Architect and maintain automated evaluation pipelines that measure agent quality across capabilities and product surfaces.
- • Translate agent capabilities into explicit success criteria — defining pass, partial-pass, and failure conditions for complex multi-step tasks.
- • Build representative gold datasets and regression suites covering common workflows, ambiguous requests, long-tail behavior, edge cases, and adversarial scenarios.
- • Define and track metrics spanning task success, partial completion, tool-selection accuracy, tool-use correctness, instruction adherence, factual consistency, user corrections, latency, cost, and reliability.
- • Design deterministic graders, model-based graders, and human-review processes; calibrate LLM-as-a-judge systems and measure false positives, false negatives, variance, and grader agreement.
- • Analyze traces, tool calls, model outputs, user context, and production outcomes to identify root causes and build a useful failure taxonomy.
- • Compare models, prompts, tools, and capability implementations using rigorous offline experiments and production evidence.
- • Build dashboards, reports, and release-quality signals that make evaluation results clear and actionable for engineering, product, and leadership.
- • Partner with capability engineers to recommend improvements and verify that fixes raise quality without unacceptable regressions in cost, latency, or reliability.
- • Must-haves:
- • 5+ years of experience in data science, machine learning, or analytics roles — with demonstrated focus on evaluation systems, metrics frameworks, or quality measurement for production systems.
- • Proven experience designing and implementing evaluation frameworks, grading systems, and metrics for ML or AI systems in production.
- • Production-quality Python and SQL proficiency; ability to build automated data pipelines and analysis code at scale.
- • Deep expertise in evaluation methodology: success criteria definition, dataset construction, metric selection, and distinguishing meaningful benchmarks from misleading ones.
- • Strong statistical and experimental design skills: sampling, variance, uncertainty quantification, bias detection, confounding, and significance testing for non-deterministic systems.
- • Experience with ground-truth data development: labeling guideline design, annotation quality control, ambiguity resolution, and dataset maintenance as product behavior evolves.
- • Working knowledge of LLM behavior, model-based graders, tool use, retrieval systems, multi-step execution, partial completion, and real-world failure modes of language model systems.
- • Analytical debugging ability: connecting quantitative patterns to individual system traces and identifying failure origins across model, prompt, context, tools, data, and application logic.
- • Experience communicating evaluation results, methodology, uncertainty, and trade-offs to both technical and non-technical stakeholders.
- • Nice to have:
- • Experience with LLM-as-a-judge systems, calibration, and grader agreement measurement.
- • Prior work on evaluation or benchmarking platforms for AI systems.
- • Experience with agentic systems, multi-step task execution, or tool-use evaluation.
- • Background working on customer-facing consumer software or production ML systems with real-world user outcomes.
🎯 Requirements
- • [5+ years of experience in data science, machine learning, or analytics roles — with demonstrated focus on evaluation systems, metrics frameworks, or quality measurement for production systems.]
- • [Proven experience designing and implementing evaluation frameworks, grading systems, and metrics for ML or AI systems in production.]
- • [Production-quality Python and SQL proficiency; ability to build automated data pipelines and analysis code at scale.]
🏖️ Benefits
- • [Competitive compensation with market rates for senior applied data science roles at early-stage AI startups.]
- • [Visa sponsorship is not available for this role.]
Skills & Technologies
See exactly how your profile matches this role — strengths, skill gaps, and what to do about them.
About Clera
Clera, Inc. operates as an AI Talent Agent, leveraging artificial intelligence to enhance and streamline various aspects of the talent acquisition and management process. The company's core offering appears to center on an AI-powered platform designed to connect skilled individuals with suitable opportunities or to assist organizations in efficiently sourcing and evaluating candidates. While comprehensive details regarding specific features, target industries, or the scale of its operations are not yet publicly available, the current online presence indicates that Clera's services are actively being prepared for launch. This suggests an upcoming introduction of innovative AI solutions aimed at optimizing the ecosystem where talent meets demand.
Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.
Newsletter
Weekly remote jobs and featured talent.
No spam. Only curated remote roles and product updates. You can unsubscribe anytime.
Similar Opportunities
3 months ago
Hangar Aviation Technologies, Inc.
30 days ago
Singular Intelligence Ltd.
2 months ago
