Job Overview
Location
San Francisco, California
Job Type
Full-time
Category
Software Engineering
Date Posted
June 21, 2026
Full Job Description
đź“‹ Description
- • Define clear, reusable evaluation frameworks and rubrics to measure what "good" looks like in Notion’s AI-powered experiences, focusing on helpfulness, trust, tone, control, and transparency.
- • Run recurring longitudinal and feature-specific qualitative studies, including side-by-side comparisons and human-in-the-loop evaluations, to assess experience quality over time and identify regressions or shifts in user expectations.
- • Anchor AI evaluations in real user workflows by studying the full interaction journey—goal setting, delegation, review, iteration—rather than isolated feedback, ensuring insights reflect actual jobs-to-be-done and user intent.
- • Identify failure modes and recovery behaviors across AI systems, including model behavior, UI, and integrations, and translate findings into actionable guidance for guardrails, fixes, and prioritization.
- • Collaborate with Product, Design, Engineering, and Data Science teams to operationalize evaluation processes, aligning on use cases and building scalable loops such as human-in-the-loop reviews and calibration of automated LLM-judge approaches against human judgment.
- • Translate qualitative user insights into concrete scoring guidelines and observable metrics for "soft" dimensions like trust, clarity, and usefulness, enabling consistent application across teams and over time.
- • Design and execute research methods tailored to each question, including interviews, benchmarking, surveys, and experiments, while prioritizing ruthlessly and balancing scrappy iteration with deep-dive analysis.
- • Communicate research findings effectively to diverse stakeholders, tailoring storytelling to connect insights to business outcomes and drive product change through clear, actionable artifacts.
- • Maintain fluency in AI products and systems, reasoning about how model behavior, uncertainty, and system constraints shape user experience, with hands-on experience evaluating LLMs, agents, and generative UI/workflow automation.
- • Work in-person on Mondays, Tuesdays, and Thursdays from Notion’s San Francisco or New York City office, as these are Anchor Days for collaborative thinking and building.
- • Contribute to a culture of craft and human-centered work, where AI is used as a true collaborator to enhance productivity and reduce fragmentation in knowledge work.
🎯 Requirements
- • 5+ years of industry experience in UX research
- • Ability to operationalize qualitative insights into measurable rubrics and metrics for subjective dimensions like trust and usefulness
- • Proven experience evaluating AI-enabled products (LLMs, agents, generative UI, workflow automation) and collaborating with Data Science/ML teams on evaluation strategy
- • Strong proficiency in both qualitative and quantitative UX research methods, with the ability to synthesize findings into actionable guidance
- • Clear communication skills to align cross-functional teams around shared definitions of quality and drive follow-through on research insights
🏖️ Benefits
- • Competitive base salary range of $196,000–$230,000 per year for roles based in San Francisco or New York City
- • Equity compensation
- • Comprehensive benefits package
- • In-office collaboration on Mondays, Tuesdays, and Thursdays (Anchor Days)
- • Opportunity to work at the forefront of AI-augmented knowledge work with a mission-driven team
- • Access to AI research and observability tooling (e.g., Dovetail, Listen Labs, Maze, Outset, Braintrust)
Skills & Technologies
See exactly how your profile matches this role — strengths, skill gaps, and what to do about them.
About Notion Labs, Inc.
Notion Labs, Inc. develops a unified workspace platform that combines documents, databases, kanban boards, calendars, and wikis. Headquartered in San Francisco, the company offers collaborative tools for teams and individuals to plan, write, organize, and share knowledge. The software integrates with third-party services and supports real-time editing, templates, and permissions management. Founded in 2013, it serves global users across education, technology, and enterprise sectors, aiming to replace fragmented productivity apps with a single, modular environment.
Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.
Newsletter
Weekly remote jobs and featured talent.
No spam. Only curated remote roles and product updates. You can unsubscribe anytime.
Similar Opportunities
Crowdtilt, Inc.
2 months ago
Datum Network Inc.
30 days ago
TrueLogic Company
2 months ago
TrueLogic Company
2 months ago