Member of Technical Staff - Training Platform

Prime Intellect, Inc.

Job Overview

Location

San Francisco

Job Type

Full-time

Full Job Description

📋 Description

• Own the design and operation of Kubernetes-based training and inference orchestration across multi-cluster, multi-cloud GPU fleets.
• Build and maintain Helm charts that compose trainers, inference servers, environment servers, and supporting services into reproducible 'Training stacks'.
• Develop the Python control-plane agents that watch pods, report run state to the platform, and keep clusters in sync.
• Implement scheduling and autoscaling for heterogeneous hardware (H100/H200/B200) using KEDA, LeaderWorkerSet, taints/tolerations, and gang scheduling.
• Run a tight GitOps workflow - every change ships through PRs, Helm values, and CI.
• Build node-local model caches, checkpoint pipelines, and shared storage for fast cold starts.
• Operate the observability stack (Prometheus, Grafana, Loki, DCGM) and make GPU cluster debugging fast.
• Build the developer-facing surfaces for hosted training: job submission, live run monitoring, logs, metrics, model/adapter management, comparisons.
• Develop FastAPI backend services and REST APIs that bridge the platform to running clusters.
• Build real-time monitoring and debugging tools (streaming logs, step-level metrics, failure analysis).
• Ship product UI in Next.js / React / TypeScript with shadcn, Tailwind, tRPC, and TanStack Query.
• Interface with the RL trainer, inference servers, and environment servers running inside our clusters.
• Productize new training capabilities (new model architectures, RL algorithms, modes).

🎯 Requirements

• Strong working knowledge of the modern AI stack - open model families, finetuning techniques (LoRA, QLoRA, full FT, RLHF/RLAIF), inference engines (vLLM, SGLang, TensorRT-LLM).
• Familiarity with GPU hardware tradeoffs (H100 / H200 / B200, NVLink, interconnects, memory hierarchy) and what they mean for training and inference workloads.
• Understanding of distributed training fundamentals (data/tensor/pipeline/expert parallelism, NCCL, multi-node scheduling).
• Awareness of what's happening at the frontier - new models, training methods, infra patterns - and the ability to translate that into product decisions.
• Strong Kubernetes operations experience - Helm, CRDs, operators, KEDA, gang scheduling, GPU operator.
• Comfortable debugging real production clusters (kubectl, pod lifecycle, node issues, networking).
• Cloud platform experience (GCP preferred - GCS, GKE, Cloud Run, Cloud Tasks).
• Infrastructure automation (Helm, Terraform, Ansible) and a GitOps mindset.
• Observability: Prometheus, Grafana, Loki, OpenTelemetry, DCGM.
• Linux fundamentals: networking, namespaces, performance tuning.
• Strong Python backend development (FastAPI, async, SQLAlchemy).
• Comfortable building Python control-plane agents that talk to Kubernetes APIs.
• Modern frontend development (TypeScript, React/Next.js, Tailwind, shadcn) - enough to ship product surfaces end-to-end.
• REST and tRPC API design.
• Experience building developer tools, dashboards, and live-monitoring UIs.

🏖️ Benefits

• Cash compensation $150K–$300K with significant equity.
• Flexible work arrangement (remote or San Francisco office).
• Full visa sponsorship and relocation support.
• Professional development budget for courses and conferences.
• Regular team off-sites and conference attendance.
• Opportunity to shape the future of decentralized AI development.

Skills & Technologies

Python

JavaScript

TypeScript

React

Next.js

Senior

Remote

$150k-300k

Ready to Apply?

Apply Externally

You will be redirected to an external site to apply.

AI Job Fit Analysis

Pro

See exactly how your profile matches this role — strengths, skill gaps, and what to do about them.

Prime Intellect, Inc.

Visit Website

About Prime Intellect, Inc.

San Francisco–based startup building decentralized AI infrastructure that lets researchers pool compute and data to collaboratively train large models. Founded in 2023, the company offers open-source protocols and cloud orchestration tools that aggregate GPUs across providers, coordinate distributed training, and cryptographically verify contributions so participants share ownership and future rewards of the resulting models.

View Company Profile

Get more remote jobs like this

Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.

Weekly remote jobs and featured talent.

No spam. Only curated remote roles and product updates. You can unsubscribe anytime.