Member of Technical Staff - Evaluations

📋 Description

• • Join ReflectionAI Inc. as a Member of Technical Staff - Evaluations, a pivotal role at the forefront of developing open superintelligence. Our mission is to democratize access to advanced AI models, making them available to a diverse range of users, from individuals and agents to large enterprises and even nation-states. You will be an integral part of a world-class team, comprised of AI researchers and seasoned company builders who have previously contributed to groundbreaking work at institutions like DeepMind, OpenAI, Google Brain, Meta, Character.AI, and Anthropic.
• • In this role, you will be instrumental in shaping the future of AI evaluation. Your primary responsibility will be to conduct rigorous and critical comparative analyses of AI model capabilities. This deep dive into model performance will be crucial for advancing our collective understanding of what these models can achieve and where their limitations lie. You will be at the heart of identifying subtle improvements and significant breakthroughs, providing the data-driven insights necessary for our research and development efforts.
• • A key aspect of your work will involve building and continuously refining sophisticated evaluation systems and processes. These systems are designed to create tight, efficient feedback loops. This means ensuring that insights derived from data analysis, evaluation results, and observed model behavior are rapidly and effectively communicated back to the teams responsible for model development. This iterative process is fundamental to accelerating model improvement and ensuring alignment with desired outcomes.
• • You will be tasked with developing generalizable evaluation frameworks. These frameworks will go beyond simple performance metrics to capture the nuanced aspects of AI capabilities that truly matter. This includes assessing reasoning abilities, ensuring alignment with human values and intentions, and measuring the overall usefulness of our models in practical applications. The goal is to create a comprehensive and adaptable system that can evolve alongside our models.
• • Collaboration is at the core of this role. You will work closely with our pre-training, post-training, and applied AI teams. Your ability to translate complex evaluation insights into actionable recommendations for model improvements will be highly valued. This cross-functional collaboration ensures that our evaluation efforts directly inform and drive the development roadmap, leading to more capable and reliable AI systems.
• • We encourage you to push the boundaries of what is currently measurable in AI evaluation. This involves exploring and implementing a wide range of evaluation methodologies. You will work with everything from synthetic evaluations designed to test specific capabilities under controlled conditions, to leveraging human feedback to gauge subjective quality and alignment, and analyzing real-world interaction data to understand performance in diverse and unpredictable environments.
• • This is an opportunity to define the cutting edge of AI evaluation. You will be working in a dynamic, fast-paced startup environment where your initiative and drive for impact are paramount. We are looking for individuals who are excited by the prospect of working in a new frontier lab, actively defining how we measure progress and accelerate the development of increasingly capable AI models. Your contributions will directly shape the trajectory of our open superintelligence initiatives.
• • The ideal candidate is highly collaborative, possesses exceptional attention to detail, and is intrinsically motivated by the process of building robust feedback loops that lead to tangible model improvements. If you are passionate about advancing AI and thrive in an environment that values innovation and rapid iteration, this role offers a unique opportunity to make a significant impact.