Senior Software Engineer, Engine & Distributed Systems

Stack AI Inc.

Job Overview

Location

SF Office - 171 2nd, 4th floor

Job Type

Full-time

Full Job Description

📋 Description

• Own the execution engine that powers every AI agent on Stack AI’s no-code platform, responsible for runtime, scheduling, and sub-agent parallelization across all customer workflows.
• Design and implement durable execution features including checkpointing, resumption, and failure recovery to ensure agents survive crashes, restarts, and network outages while picking up exactly where they left off.
• Shape the core execution model by defining how work is scheduled, queued, and transitioned between synchronous and asynchronous modes to maintain platform correctness and responsiveness under increasing load.
• Engineer for extreme scale and reliability by enforcing strict health targets for worker freshness, deploy safety, and drain time, while maintaining low latency and high throughput as agent volume grows exponentially.
• Build and maintain the engine’s openness to the ecosystem by enabling seamless integration of new agent harnesses, orchestration frameworks, and model capabilities into the runtime without compromising stability.
• Drive end-to-end ownership of the engine’s production behavior, from initial design and testing to deployment, monitoring, and incident response, ensuring zero data loss and deterministic execution under all conditions.
• Collaborate with product and infrastructure teams to align engine capabilities with customer use cases, including chatbots, document-processing pipelines, and database Q&A tools that run for seconds or days.
• Optimize the engine’s performance and resource efficiency to reduce operational costs while maintaining high availability and throughput across distributed worker pools.
• Implement robust retry mechanisms, concurrency controls, and fault-tolerant patterns to handle partial failures, network partitions, and resource contention without degrading user experience.
• Contribute to architectural decisions around state management, message passing, and coordination primitives that underpin the engine’s ability to handle complex, long-running, fan-out workflows with sub-agents.
• Maintain clear documentation and observability tooling for the engine’s internal state, metrics, and logs to enable rapid debugging and proactive issue detection in production.
• Participate in on-call rotations to respond to critical incidents affecting the engine’s availability or correctness, with a focus on root cause analysis and prevention of recurrence.
• Advocate for best practices in distributed systems design, including idempotency, eventual consistency, and state machine modeling, across engineering teams.
• Evaluate and integrate emerging technologies in AI agent runtimes, such as tool-calling, streaming outputs, and sub-agent orchestration frameworks, to enhance the engine’s capabilities.
• Work closely with backend and frontend teams to ensure the engine’s API and behavior align with the visual, drag-and-drop interface used by non-technical users to build AI workflows.

🎯 Requirements

• 5+ years building backend systems in production, with real depth in distributed systems.
• Hands-on experience with durable execution or workflow orchestration (Temporal, Cadence, Airflow, or equivalent), with a way of thinking rooted in idempotency, state machines, and failure recovery.
• Strong command of concurrency, queueing, retries, and fault tolerance under load.
• Strong in Python and modern backend frameworks (FastAPI or similar), with sound database fundamentals (Postgres or similar).
• You're drawn to the correctness problems that everything else quietly depends on.
• Distributed systems is broad. If you're strong on most of this and excited to grow into the rest, we'd like to hear from you, even if you don't check every box.

🏖️ Benefits

• Join a lean, high-impact team and own the engine that every customer's agents run on.
• Your work ships fast and is felt across the whole product.
• Stack AI is an equal opportunity employer that celebrates diversity and is committed to creating an inclusive environment for all employees.

Skills & Technologies

Python

FastAPI

PostgreSQL

REST

Senior

Onsite

Ready to Apply?

Apply Externally

You will be redirected to an external site to apply.

AI Job Fit Analysis

Pro

See exactly how your profile matches this role — strengths, skill gaps, and what to do about them.

Stack AI Inc.

Visit Website

About Stack AI Inc.

Stack AI Inc. provides a low-code platform for building, deploying and scaling AI workflows and applications. Its cloud service offers drag-and-drop builders, pre-built templates, vector databases, model orchestration and enterprise-grade security, enabling teams to integrate large language models and data sources into chatbots, assistants and process automations without deep ML expertise.

View Company Profile

Get more remote jobs like this

Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.

Weekly remote jobs and featured talent.

No spam. Only curated remote roles and product updates. You can unsubscribe anytime.