LLM Ops Evaluations

📋 Description

• • Are you passionate about the intricate world of Large Language Models (LLMs) and driven to ensure their real-world performance meets and exceeds expectations? Sintra is seeking a highly skilled and motivated LLM Ops Specialist to take ownership of the quality and effectiveness of our AI "teammates." In a landscape where 50,000 small business owners rely on Sintra daily, our AI helpers are not just tools; they are integral components that reply to customers, manage social media, analyze data, and perform a myriad of other crucial tasks. For many of our users, these AI assistants represent the first tangible and effective help they've ever had in running their businesses. We've built something that resonates deeply with our users, and now we're focused on elevating it to new heights of excellence.
• • This is a foundational role where you will be instrumental in shaping the future of AI quality at Sintra. You will be responsible for designing and implementing our comprehensive evaluation framework, acting as the primary technical liaison with leading LLM providers, and driving the continuous improvement of LLM outputs across our entire application. Your expertise will directly impact the reliability, accuracy, and overall helpfulness of our AI helpers, ensuring they continue to feel like true teammates to our users.
• • Your core mission will be to build and refine the systems that rigorously assess and guarantee the performance of our AI helpers. This involves a deep dive into prompt engineering, meticulous evaluation processes, the curation of robust datasets, and strategic model selection. You will be at the forefront of defining what "quality" means for AI-powered products at a rapidly scaling company, translating complex technical capabilities into tangible benefits for small business owners.
• • Key responsibilities will include:
• • Owning the end-to-end quality assurance of all AI-generated outputs across the Sintra platform. This encompasses everything from initial prompt design to final output validation, ensuring consistency, accuracy, and adherence to brand voice and user expectations.
• • Designing, building, and iterating on our comprehensive evaluation framework. This will involve developing automated testing suites to catch regressions, establishing effective human review loops for nuanced assessments, and creating robust quality scoring mechanisms to quantify performance.
• • Creating, versioning, and meticulously optimizing prompts for a wide array of use cases. You will be responsible for ensuring that each AI helper is equipped with the most effective instructions to perform its specific tasks with precision and efficiency.
• • Building and maintaining high-quality, representative test datasets. These datasets are critical for identifying performance degradation and preventing regressions before they impact our users, acting as a vital safety net for our AI functionalities.
• • Collaborating closely with engineering teams to integrate evaluation systems and automate quality checks, ensuring that quality is a built-in aspect of our development lifecycle, not an afterthought.
• • Serving as the primary technical point of contact with leading AI labs and LLM providers. This involves understanding their roadmaps, providing feedback on model performance, and advocating for features that enhance our ability to deliver high-quality AI experiences.
• • Developing strategies for handling the complexity of hundreds of potential use cases and user customization options, ensuring our evaluation framework is scalable and adaptable.
• • As Sintra scales, you will have the opportunity to hire and lead a team of talented prompt engineers and evaluation specialists, fostering a culture of quality and continuous improvement within the AI team.
• • Staying abreast of the latest advancements in LLM technology, evaluation methodologies, and best practices in MLOps to continuously enhance our AI quality standards.
• • This role is ideal for an individual who has a proven track record of shipping LLM-powered products at scale, possesses a strong systems-thinking mindset, and is eager to define the standards for AI quality in a fast-growing organization. If you are excited by the prospect of shaping the intelligence behind a product that genuinely empowers small businesses, this is your opportunity.