llm-d 1.0 Kubernetes inference stack for TPU and GPU

What changed

Google and a coalition of infrastructure partners released llm-d 1.0 on October 20, 2026 at PyTorch Conference North America. This stable version of an open-source distributed inference stack is built for Kubernetes. The project, founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, delivers production-grade capabilities that the team describes as day 2 operational reliability.

The 1.0 release includes multi-tenant quality of service with priority and fairness-aware flow control, hardened v1 APIs validated across multiple clouds and accelerators, zero-downtime upgrades with router high availability and hot standby capability, and session-aware routing optimized for agent workloads. The stack now supports asynchronous inference that pulls jobs from Google Cloud Pub/Sub or Redis, plus expanded native PyTorch support on TPUs through TorchTPU integration with vLLM and SGLang.

The llm-d project is governed by the Cloud Native Computing Foundation as a Sandbox project, accepted on March 12, 2026. It has more than 500 contributors from over 25 organizations and carries an Apache 2.0 license. Red Hat, IBM and Google serve as lead maintainers.

Why it matters

Teams deploying large language models at scale encounter a fundamental shift in constraints. As model training complexity recedes, the challenge becomes the cost and operational complexity of serving inference reliably to many agents running in parallel. Agentic workloads create unpredictable traffic bursts and demand constant access to large cached context, placing stress on both accelerator memory and CPU capacity.

llm-d 1.0 addresses this bottleneck by coordinating vLLM or SGLang model servers into a single managed pool through Kubernetes. The stack runs on NVIDIA, AMD or Intel GPUs, or Google TPUs, on any Kubernetes cluster. It monitors which replicas hold cached prefixes for each request context and routes new queries to the replica that already contains that context in memory, eliminating wasted computation and memory allocation.

For teams managing multiple agents or high-traffic inference endpoints, this changes the deployment economics. Reported results indicate cost reductions of five to 10 times for context-heavy agentic work and three times greater output throughput compared to unoptimized Kubernetes inference setups. Snap has achieved prefix cache hit rates of 75 to 80 percent. Zenteiq reports throughput scaling from 1.47 billion to 5.97 billion daily tokens per 1,000 TPU v6e chips. An IBM demo served 3,000 concurrent coding agents at 5 to 10 times below API pricing.

The production-grade release enables enterprise teams to deploy llm-d with confidence that operational challenges including failover, scaling, quota enforcement and live upgrades function reliably across multiple clusters and regions.

What to test

Before adopting llm-d 1.0, teams should verify performance claims and assess fit for their specific workload:

Benchmark context

Published performance figures originate from specific hardware configurations and model sizes. The 85% median reduction in time-to-first-token and 2.4x throughput improvement were measured on Ironwood TPU with prefix-aware routing and host memory offload on coding workloads. Evaluate these features on your own accelerators and model sizes. The 17,000-node GKE benchmark result showing 2.9x throughput improvement across 3 clusters represents an internal Google Cloud measurement; validate scaling behavior in your own cluster topology.

Cache hit rate variability

Prefix cache hit rate gains depend heavily on request patterns. Gemini doubled hit rates from 35% to 70%, but your agents may exhibit different context reuse characteristics. Deploy a staged rollout with the router positioned in front of an existing vLLM cluster, then measure cache hits against your actual traffic before committing to tiered key-value offload or disaggregation strategies.

Accelerator utilization limits

According to published results, reinforcement learning workloads achieved above 70% GPU duty cycle using time-slicing, rising from 40%. This still leaves roughly 30% idle time. Determine whether this utilization level meets your efficiency requirements, or whether your batch patterns could support higher utilization without added architectural complexity.

Deployment staging and tooling

Google recommends a phased approach beginning with prefix-aware routing, then adding flow control, tiered offload and disaggregation. Verify that Helm charts work with your Kubernetes version and container network interface. Test GKE Inference Gateway, the managed router service on Google Cloud, in a non-production cluster before wider rollout.

Failover and multi-region behavior

llm-d supports zero-downtime upgrades and spans clusters and regions with global failover. Test this functionality in your non-production environment to understand failover timing and whether it aligns with your service level agreements.

Common questions

What does llm-d do?

llm-d is a Kubernetes-native distributed inference serving stack that coordinates multiple model servers such as vLLM or SGLang into one logical pool. It tracks cached model context on each replica and routes new requests to the server that already holds that context in memory, reducing redundant computation and accelerator memory use. It also manages request priorities, offloads cache to host memory or storage when accelerator memory fills, and separates prefill operations, which are compute-heavy, from decode operations, which are memory-bound, into independent pools.

How to accelerate LLM inference?

llm-d offers several techniques. Prefix-aware routing sends requests to servers that already have the input context cached, eliminating redundant computation. Tiered key-value cache offload moves cached key-value pairs from accelerator memory to host memory, then to SSD or managed storage when needed. Prefill-decode disaggregation runs input processing and output generation on separate pools tuned for their workload characteristics. Multi-tenant flow control prioritizes latency-sensitive requests over bursty agents, preventing starvation during traffic spikes.

How does llm-d differ from vLLM?

vLLM is a single-node model serving engine optimized for throughput and latency. llm-d is a distributed inference orchestration layer that runs on top of vLLM or SGLang and Kubernetes. It coordinates multiple vLLM instances across many nodes, handles request routing based on cached context, manages memory tiers, and enforces multi-tenant quality of service. Teams typically deploy vLLM as the per-node serving engine and llm-d as the cluster-wide orchestrator.

The conclusion

llm-d 1.0 represents the first mature release of a distributed inference orchestrator built for production agentic AI workloads. The release advances the project from early capabilities toward operational reliability. Teams running high-volume inference with complex agent patterns will find value in testing the cache-aware routing and disaggregated serving architecture, particularly if they aim to reduce inference costs or maximize hardware efficiency.

The benchmark results are strong but context-specific. The real value lies not in the individual performance gains, but in the architectural shift from single-node model servers to a cluster-wide inference pool that avoids redundant computation and memory waste. For teams building multi-agent systems or running long-context workloads, this efficiency improvement is meaningful.

Watch for adoption trends among hyperscalers and frontier model labs. If llm-d becomes the standard approach for distributed Kubernetes inference, ecosystem support in related tools and platforms will expand. For now, begin with a non-production pilot using the published Helm charts and measure cache hits and latency against your baseline vLLM deployment.

Related reading

• DeepInfer: DeepSeek open-sources inference engine
• Voxtral 3 voice agent SDK: 50+ languages and emotional speech
• Exa 3.0 search API: Tops Perplexity and Brave on 8 benchmarks