Business Analyst with AI evaluation skills

📋 Description

• • Gramian Consultancy, a specialized boutique firm dedicated to providing top-tier IT professional services and engineering talent solutions, is seeking a highly motivated and detail-oriented Business Analyst to join our team. With a robust foundation in software engineering and leadership principles, we pride ourselves on our ability to connect companies with exceptional professionals who are a perfect fit for their unique needs and organizational culture. This role is specifically designed to support critical evaluation and annotation workflows, with a strong emphasis on ensuring quality, accuracy, and the integrity of structured reasoning within AI-generated outputs.
• • As a Business Analyst with AI evaluation skills, you will play a pivotal role in a dynamic and evolving field. Your primary responsibility will be to conduct rigorous side-by-side (SxS) evaluations of AI-generated content, meticulously comparing different outputs to assess their quality, accuracy, and overall relevance. This involves a deep dive into the nuances of AI performance, requiring a keen eye for detail and a systematic approach to analysis.
• • A core component of this role is performing thorough fact-checking. You will be tasked with validating information presented in AI outputs against reliable sources, identifying any inconsistencies, inaccuracies, or potential biases. This critical step ensures that the information being processed and evaluated meets the highest standards of truthfulness and reliability.
• • Following established evaluation guidelines and quality standards will be paramount. You will need to consistently adhere to predefined protocols and frameworks to ensure uniformity and objectivity in your assessments. This adherence is crucial for generating reliable data that can be used to improve AI models and processes.
• • You will be expected to provide detailed, well-structured justifications to support every evaluation decision. This means not only identifying issues but also clearly articulating the reasoning behind your assessment, referencing specific evidence and criteria. This ability to communicate complex findings in a clear, concise, and objective manner is a hallmark of this position.
• • The role requires proficiency in using various digital tools, including browser-based applications and extensions, to facilitate analysis and review tasks. You will leverage these tools to efficiently gather information, perform comparisons, and document your findings. Familiarity with these digital environments will streamline your workflow and enhance your productivity.
• • In addition to standard digital tools, you may be required to execute simple command-line instructions. This is not a programming-intensive task but rather a functional requirement to support specific evaluation workflows. Basic comfort with command-line interfaces will be beneficial.
• • Documenting your findings clearly and communicating insights to stakeholders, including project managers and AI development teams, will be an essential part of your contribution. Your ability to translate complex analytical results into actionable feedback will directly impact the improvement of AI systems.
• • This is a contractor assignment with a commitment of 6-8 hours per day. A key aspect of the role is the requirement for an overlap of 3 hours with Pacific Standard Time (PST), ensuring effective collaboration with global teams. The contract duration is for 2 months, offering a focused and intensive engagement.
• • The selection process includes a 60-minute technical test specifically designed to assess your ability to compare and evaluate AI outputs, simulating the core tasks of the role. This test is a crucial step in identifying candidates with the necessary analytical acumen and attention to detail.
• • This position offers a unique opportunity to work remotely, providing flexibility and the ability to manage your work from anywhere. Furthermore, you will gain invaluable experience working on cutting-edge AI projects, collaborating with leading technology companies and contributing to the advancement of artificial intelligence.