Senior Technical Program Manager, Human-in-the-Loop Operations

Roblox Corporation

Job Overview

Location

San Mateo, CA, United States

Job Type

Full-time

Full Job Description

📋 Description

• At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device.
• A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone.
• As Senior Technical Program Manager for the Human-in-the-Loop (HITL) Operations, you will own the end-to-end lifecycle of Roblox's AI data labeling and model evaluation ecosystem.
• This is a high-leverage, high-ownership role at the intersection of Research, Engineering, and Operations.
• You will scale our data operations that is handling million of items evaluated annually today with projection of 4x the current volume by 2030, managing a multi-million annual budget and a distributed workforce of 100+ remote contractors — all while driving the platform evolution from manual workflows to AI-assisted, LLM-accelerated operations.
• This is a rare opportunity to build the data infrastructure that directly determines the quality of Roblox's AI models across creator tools, content discovery, in-experience AI, and 3D generative content — at the exact moment when human judgment is the critical ingredient for getting these models right.
• You will lead data programs end-to-end. Own the full lifecycle of labeling and model evaluation workflows across Roblox's AI teams — from translating ambiguous ML requirements into structured annotation tasks, to overseeing contractor execution, quality review, and dataset delivery to model teams.
• You will define and maintain ground truth. Develop and iterate on labeling guidelines, annotation schemas, and quality frameworks tailored to Roblox's unique data types: 3D mesh quality, texture coherence, search relevance, game novelty detection, NPC behavior evaluation, and AI-generated Luau code assessment.
• You will manage a multi-vendor, distributed workforce. Oversee remote contractor teams, managing workforce allocation, productivity SLAs, quality calibration, and budget forecasting serving across multiple teams with Roblox.
• You will drive the shift to AI-assisted workflows. Partner with Engineering to evaluate and implement LLM-based pre-labeling, LLM-as-a-Judge evaluation, and automated QA routing — reducing FTE coordination overhead and scaling throughput proportional to our annual volume growth.
• You will partner cross-functionally across Roblox AI. Work closely with ML Engineers, Data Scientists, and Product Managers to translate model development needs into concrete, executable data programs, and communicate program status and quality trends to senior leadership.
• You will own operational excellence and KPIs. Define and track SLAs across throughput, inter-annotator agreement, cost per label, and data quality metrics. Surface risks, resolve bottlenecks, and drive continuous process improvement.
• You will shape platform and tooling strategy. Contribute to Roblox's next-generation labeling and evaluation platform strategy — including vendor pilots, internal platform improvements, and the transition to a hybrid human-AI collaborative workflow model.

🎯 Requirements

• 5+ years of experience in Technical Program Management, Data Operations, or Operations Management within an AI/ML or data-intensive environment.
• Deep hands-on experience with data labeling, annotation, or model evaluation — including designing annotation schemas, writing labeling guidelines, and managing quality control at scale.
• Proven experience managing external vendor relationships and distributed contractor workforces, including workforce planning, quality oversight, and budget management.

🏖️ Benefits

• Annual Salary Range: $226,400 - $277,350 USD
• Equity compensation
• Benefits as described on this page

Skills & Technologies

Python

Senior

Remote

Ready to Apply?

Apply Externally

You will be redirected to an external site to apply.

AI Job Fit Analysis

Pro

See exactly how your profile matches this role — strengths, skill gaps, and what to do about them.

Roblox Corporation

Visit Website

About Roblox Corporation

Roblox Corporation develops and operates the Roblox online platform where users create, share, and play games and experiences built by its global community. The company offers a game engine, cloud services, and social infrastructure enabling user-generated 3D worlds and virtual economies. Revenue comes from virtual currency sales, advertising, and licensing. Founded in 2004 and headquartered in San Mateo, California, Roblox serves millions of daily active users worldwide across PC, mobile, and console devices.

View Company Profile

Get more remote jobs like this

Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.

Weekly remote jobs and featured talent.

No spam. Only curated remote roles and product updates. You can unsubscribe anytime.