Job Overview
Location
Singapore
Job Type
Full-time
Category
Data Science
Date Posted
April 6, 2026
Full Job Description
đź“‹ Description
- • AI QA Trainer role focused on evaluating and improving large language models through rigorous testing, safety checks, and quality assurance to ensure enterprise-grade AI reliability and performance.
- • Day-to-day responsibilities include conversing with models on real-world scenarios, verifying factual accuracy and logical soundness, designing and executing test plans and regression suites, building evaluation rubrics, capturing error traces with root-cause analysis, and suggesting improvements to prompt engineering, guardrails, and evaluation metrics.
- • Work involves collaboration on adversarial red-teaming, test automation using Python/SQL, and dashboarding to track quality trends over time, contributing to the hardening of model reasoning and reliability.
- • Opportunity to deepen expertise in LLM safety, multilingual and domain-specific testing, grounding verification, bias/fairness auditing, and evaluation metric design while shaping the quality backbone of next-generation AI systems.
🎯 Requirements
- • Bachelor’s, master’s, or PhD in computer science, data science, computational linguistics, statistics, or a related field.
- • Experience with shipped QA for ML/AI systems, safety/red-team work, or test automation frameworks (e.g., PyTest).
- • Hands-on experience with LLM evaluation tooling such as OpenAI Evals, RAG evaluators, or Weights & Biases (W&B).
- • Skills in evaluation rubric design, adversarial testing/regression testing at scale, bias/fairness auditing, grounding verification, and prompt/system-prompt engineering.
- • Proficiency in test automation using Python and/or SQL.
- • Strong metacognitive communication skills with ability to clearly document and "show your work".
🏖️ Benefits
- • Competitive hourly pay range of $6 to $65, with final rate based on experience, expertise, and location.
- • Fully remote work arrangement allowing flexibility in work location and schedule.
- • Contract-based engagement ideal for freelancers seeking independent, project-driven AI QA work.
- • Opportunity to contribute to cutting-edge AI safety and reliability efforts at the forefront of LLM development.
Skills & Technologies
About Agency Analytics Inc.
Agency Analytics provides cloud-based marketing analytics and reporting software for digital agencies and freelancers. The platform integrates data from Google, Facebook, LinkedIn, and other channels, offering white-label dashboards, automated reporting, and SEO tools. Founded in 2009 and headquartered in Toronto, Canada, the company serves thousands of agencies worldwide seeking to streamline client reporting and demonstrate ROI.
Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.
Newsletter
Weekly remote jobs and featured talent.
No spam. Only curated remote roles and product updates. You can unsubscribe anytime.
Similar Opportunities
Definitive Healthcare Corporation
7 months ago
