AI Evaluation Program Manager

Twelve Labs Inc.

Job Overview

Location

San Francisco

Job Type

Full-time

Full Job Description

📋 Description

• As an AI Evaluation Program Manager at Twelve Labs, you will play a vital role in advancing the company’s mission to pioneer multimodal foundation models that comprehend video like humans do, directly contributing to the evaluation and refinement of cutting-edge AI systems.
• You will design and build robust model evaluation frameworks, manage resource allocation and timelines across data streams, collaborate with external partners to enhance data quality, establish labeling guidelines, and partner with engineering and AI model teams to align on data needs and communicate progress through analytical reports and dashboards.
• You will join the ML Data Team, which leads the full spectrum of video-language data preparation and model evaluation, operating within a well-funded, innovative AI company backed by top-tier investors including NVIDIA’s NVentures, NEA, and Radical Ventures, with headquarters in San Francisco and a global presence in Seoul.
• In this role, you will develop expertise in large-scale AI data operations, gain hands-on experience with multimodal AI evaluation, strengthen cross-functional collaboration skills, and contribute to building scalable data infrastructure that supports state-of-the-art model performance and assessment.

🎯 Requirements

• 5+ years of experience working in an AI focused data operations organization
• Proven track record designing and executing large scale data or evaluation projects, including gathering, labeling, and post-processing data
• Ability to analyze messy and complex data, identify overarching patterns, and distill findings into crisp annotation guidelines or model quality reports
• Proficiency with Python, LLMs, or other popular industry tools for automation
• Excellent communication and project management skills, with ability to support several projects simultaneously
• Foundational understanding of and interest in LLMs/VLMs and multimodal AI

🏖️ Benefits

• An open and inclusive culture and work environment
• Opportunity to work closely with a collaborative, mission-driven team on cutting-edge AI technology
• Full health, dental, and vision benefits
• Flexible PTO and parental leave policy, with office closed the week of Christmas and New Years

Skills & Technologies

Python

Pandas

Jupyter

Onsite

Ready to Apply?

Apply Externally

You will be redirected to an external site to apply.

Twelve Labs Inc.

Visit Website

About Twelve Labs Inc.

Twelve Labs builds multimodal video understanding AI. Its cloud platform transforms long-form video into vector embeddings that capture visual, audio, speech and contextual information, enabling semantic search, summarization, chaptering, moderation and analytics through a single API. Developers upload video, index it, then query in natural language or image to retrieve exact moments, generate highlights or detect unwanted content. Models are pretrained on large-scale web video, continually fine-tuned for accuracy and latency, and deployable on dedicated GPU clusters for enterprise security. Founded in 2021, the San Francisco company serves media, ed-tech, safety and e-commerce customers worldwide.

View Company Profile

Get more remote jobs like this

Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.

Weekly remote jobs and featured talent.

No spam. Only curated remote roles and product updates. You can unsubscribe anytime.