Business Analyst with AI evaluation skills

📋 Description

• • Gramian Consultancy, a distinguished boutique firm specializing in IT professional services and engineering talent solutions, is seeking a highly motivated and detail-oriented Business Analyst to join our team. With a robust foundation in software engineering and leadership, we pride ourselves on empowering companies to build exceptional, high-performing teams by meticulously matching them with professionals who align perfectly with their unique needs and cultural fit. This specific role is designed to support critical evaluation and annotation workflows, with a keen focus on ensuring the highest standards of quality, accuracy, and structured reasoning in AI-generated content.
• • As a Business Analyst with AI evaluation skills, you will play a pivotal role in the assessment of AI outputs, contributing directly to the refinement and improvement of artificial intelligence models. Your primary responsibility will involve performing rigorous side-by-side (SxS) evaluations, meticulously comparing different outputs generated by AI models (referred to as GHD in this context) and critically assessing their quality, accuracy, and overall relevance to the given prompts or tasks. This analytical process requires a sharp eye for detail and a commitment to objective assessment.
• • A core component of this role is conducting thorough fact-checking to validate the information presented in AI outputs. You will be tasked with identifying any inconsistencies, inaccuracies, or misleading statements, ensuring that the information is reliable and factually sound. This diligent approach to verification is crucial for maintaining the integrity of the AI models we are evaluating.
• • Following established evaluation guidelines and adhering to stringent quality standards will be paramount. You will need to consistently apply these frameworks to your assessments, ensuring uniformity and reliability across all evaluations. This structured approach guarantees that our feedback loop is effective and actionable for the development teams.
• • The role involves extensive use of digital tools, including browser-based platforms and extensions, to facilitate the analysis and review process. Proficiency in navigating and utilizing these tools efficiently will be essential for success. Additionally, you may be required to execute simple command-line instructions to support specific evaluation workflows, though advanced programming knowledge is not expected.
• • You will be responsible for providing detailed, well-supported justifications for every evaluation decision made. These justifications must be clear, concise, and objectively reasoned, demonstrating a deep understanding of the evaluation criteria and the nuances of the AI outputs. Effective communication of your findings and insights to relevant stakeholders, when necessary, will also be a key aspect of the role.
• • This is a contractor assignment requiring a commitment of 6-8 hours per day, with a mandatory overlap of 3 hours with the Pacific Standard Time (PST) zone. The contract duration is for 2 months, and it is a remote position open to candidates in India, Pakistan, Nigeria, Kenya, Egypt, and Ghana. Please note that as a contractor, this role does not include medical benefits or paid leave.
• • The selection process includes a 60-minute technical test specifically designed to assess your ability to compare and evaluate AI outputs, simulating the core tasks of the role. This test will provide a practical demonstration of your analytical capabilities and attention to detail in the context of AI evaluation.
• • This opportunity offers a unique chance to gain hands-on experience with cutting-edge AI technologies and contribute to the development of advanced AI systems. You will be working in a dynamic, fully remote environment, collaborating with a team dedicated to innovation and excellence in the tech industry.