Job Overview
Location
Boston
Job Type
Full-time
Category
Machine Learning Engineer
Date Posted
May 7, 2026
Full Job Description
đź“‹ Description
- • As a Machine Learning Engineer focused on distributed vLLM infrastructure in the llm-d project at Red Hat, you will be at the forefront of innovation, working to accelerate AI for the enterprise by bringing operational simplicity to GenAI deployments through open-source LLMs and vLLM.
- • You will contribute to the design, development, and testing of new features for Red Hat AI Inference, develop and maintain distributed inference infrastructure using Kubernetes APIs and operators, and build system components in Go and/or Rust to integrate with the vLLM project and manage distributed workloads.
- • You will work on KV cache-aware routing and scoring algorithms to optimize memory utilization, enhance resource utilization and fault tolerance of the inference stack, and develop and test various inference optimization algorithms.
- • You will collaborate with engineering and cross-functional teams, participate in technical design discussions, provide code reviews, and be mentored by senior team members while contributing to a culture of continuous improvement and knowledge sharing.
- • You will help shape the future of AI deployment by working on cutting-edge problems at the intersection of deep learning, distributed systems, and cloud-native infrastructure in an open-source environment.
🎯 Requirements
- • Strong proficiency in Python and/or GoLang or similar language
- • Experience with cloud-native Kubernetes service mesh technologies/stacks such as Istio, Cilium, Envoy (WASM filters), and CNI
- • Working understanding of Layer 7 networking, HTTP/2, gRPC, and the fundamentals of API gateways and reverse proxies
- • Knowledge of serving runtime technologies for hosting LLMs, such as vLLM, SGLang, TensorRT-LLM, etc.
- • Excellent written and verbal communication skills, capable of interacting effectively with both technical and non-technical team members
- • Ability to work independently in a dynamic, fast-paced environment
🏖️ Benefits
- • Comprehensive medical, dental, and vision coverage
- • Retirement 401(k) with employer match
- • Paid time off and holidays
- • Paid parental leave plans for all new parents
- • Additional benefits including employee stock purchase plan, family planning reimbursement, tuition reimbursement, transportation expense account, and employee assistance program
Skills & Technologies
About Red Hat, Inc.
Red Hat, Inc. is an American software company that provides enterprise open-source solutions, including its flagship Red Hat Enterprise Linux operating system, hybrid cloud platforms, container and Kubernetes technologies, middleware, storage, and automation tools. Founded in 1993 and headquartered in Raleigh, North Carolina, it became a subsidiary of IBM in 2019. The company supports organizations in modernizing and managing IT infrastructure through subscription-based support, training, and certification services, emphasizing security, scalability, and interoperability across hybrid and multicloud environments.
Subscribe to the weekly newsletter for similar remote roles and curated hiring updates.
Newsletter
Weekly remote jobs and featured talent.
No spam. Only curated remote roles and product updates. You can unsubscribe anytime.
Similar Opportunities
16 days ago
Heidi Health Pty Ltd
3 months ago