Principal AI Engineer

Gorgias Inc.

Job Overview

Location

Belgrade

Job Type

Full-time

Full Job Description

📋 Description

• Architect and lead the development of an internal AI evaluation platform to automate the end-to-end lifecycle of LLM agent testing, from prompt iteration to production monitoring.
• Design scalable infrastructure and tooling used by AI and engineering teams to reduce manual processes and accelerate the delivery of production-ready conversational agents.
• Partner with the Orchestration team to build robust, high-scale systems capable of running complex AI evaluations and agentic simulations.
• Serve as the AI Technical Lead for product squads, guiding teams on agent design, failure analysis, and prompting best practices to enable autonomous quality measurement.
• Define and institutionalize measurable standards for AI performance, translating non-deterministic LLM behavior into predictable, trustable engineering metrics.
• Own the feedback loop between production agent performance and evaluation datasets, ensuring real-world insights directly inform iterative improvements.
• Mentor engineers on applying rigorous software engineering principles—such as system design, observability, and maintainability—to AI and agent-based systems.
• Build deterministic wrappers and evaluation loops around probabilistic LLMs to ensure reliability, safety, and consistency in production environments.
• Develop multi-stage AI orchestrations including chained workflows, state machines, and custom runtime logic beyond simple API calls to LLMs.
• Implement and maintain systems using vector databases, semantic caching, and third-party API integrations to support scalable AI operations.
• Drive adoption of evaluation frameworks that replace vibes-based testing with data-driven, reproducible metrics for AI model performance.
• Collaborate cross-functionally with AI, ML, and product teams to align evaluation systems with business KPIs and product goals.
• Raise the engineering bar across teams by establishing best practices for monitoring, observability, and system performance in AI-driven applications.
• Enable decentralization of AI quality ownership by creating the "paved road" that allows product squads to independently measure and maintain agent quality.

🎯 Requirements

• 8+ years of engineering experience at a Staff-level or equivalent
• Proven experience architecting distributed systems and event-driven architectures
• Hands-on experience with LLMs including prompting, RAG, agent workflows, and evaluation frameworks
• Experience building scalable backend services and APIs, preferably in Python
• Demonstrated ability to move beyond simple LLM API calls to design multi-stage AI orchestrations
• Strong understanding of reliability, observability, and system performance in production AI systems

🏖️ Benefits

• 5-week vacation plus 2 weeks RTT
• 6 weeks of full remote work per year
• Paid parental leave of 16 weeks
• MacBook Pro provided
• Personal credit card for lunches via Swile
• Private health insurance provided through GAN
• Up to €700 for home workstation setup
• Up to €2000 annually for learning materials and wellness support (€1500 for job-related learning, €500 for wellness)
• 50% reimbursement of public transportation costs
• Quarterly online company-wide summits and annual offsite retreats

Skills & Technologies

Python

Kafka

Data Science

Senior

Remote

Ready to Apply?

Apply Externally

You will be redirected to an external site to apply.

AI Job Fit Analysis

Pro

See exactly how your profile matches this role — strengths, skill gaps, and what to do about them.

Gorgias Inc.

Visit Website

About Gorgias Inc.

Gorgias Inc. provides a helpdesk platform for e-commerce merchants. The cloud software centralizes customer support tickets from email, chat, social media, and voice into one dashboard. Automation rules, macros, and integrations with Shopify, Magento, and BigCommerce enable teams to resolve inquiries faster. Features include order editing, refunds, and revenue reporting tied to support interactions. Founded in 2015, the company serves over 12,000 online stores worldwide from offices in San Francisco, New York, Paris, and remote locations.

View Company Profile

Get more remote jobs like this

Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.

Weekly remote jobs and featured talent.

No spam. Only curated remote roles and product updates. You can unsubscribe anytime.