AKV cache engineering cuts agent costs 27.7%

Researchers from Tsinghua University and CUHK have released AKV, an open-source serving system that rethinks how language model agents manage their key-value (KV) cache. Instead of the standard approach of summarizing or compressing old conversation history, AKV allows agents to selectively drop and reposition KV cache entries while preserving the message text completely unchanged. The system is built on SGLang and released under an Apache 2.0 license on GitHub.

The key insight behind AKV cache engineering is that KV cache reads account for 82.4% of total API cost in long-agent runs. Rather than invalidating cached states by editing the conversation prefix, which forces expensive recomputation, AKV keeps the text trajectory unchanged while modifying only which KV entries remain visible to attention. Retained entries get reused without recalculation. This separation of concerns allows agents to maintain a richer conversation history without the cost penalty of storing it all.

What changed

Tsinghua and CUHK researchers open-sourced AKV, a cache engineering server that works alongside SGLang’s inference engine. The system combines agent-side policies for deciding what KV entries to drop and when to compact token positions with a serving backend that executes those operations while reusing compatible cached states. Testing covered GPT-OSS-120B with 5.1B active parameters, MiniMax-M2.7 with 10B active parameters out of 230B total, and MiroThinker-1.7-Mini with 3B active parameters out of 30B total.

Across benchmarks including BrowseComp-Plus, GAIA-Text, SWE-bench Verified, and xbench-DeepSearch-2510, the approach was measured against two baselines. The Standard baseline uses no summarization. Text Engineering applies aggressive summarization at 48K tokens, or 96K for MiroThinker. AKV’s default policy retains the most recent 12 tool responses and compacts token positions periodically.

The primary results show average cost falling 27.7% and average task score rising 1.7 points when compared to threshold summarization. On the strongest benchmark case, GPT-OSS-120B on SWE-bench Verified, cost per task decreased from 20.0 cents to 9.7 cents while accuracy improved from 40.0 to 43.2. AKV also achieved 21.9% cost reduction compared to Text Engineering.

Why it matters

LLM agents running long-horizon tasks like web browsing, code review, or multi-step research face substantial costs as context expands. While model-level approaches like increased efficiency prioritize throughput, AKV addresses the problem from the serving layer by allowing agents to preserve accuracy while reducing overhead.

The practical benefit is that agents can operate longer without budget constraints, and longer operations enable better performance on complex problems. The 1.7-point average accuracy improvement indicates that AKV’s selective drop approach loses less information than aggressive summarization, which can discard important details from earlier conversation turns.

AKV introduces three cache operations. Append is the standard token update. Drop removes selected KV entries from future attention without renumbering token positions. Reposition rewrites retained keys to positions 0 through r-1 using RoPE rotation, while leaving values unchanged, which avoids hidden state recomputation. In ablations, adding Reposition alone raised GPT-OSS-120B’s pass rate on BrowseComp-Plus from 55% to 62%.

The serving challenge is substantial. Identical tokens can produce different cache states depending on their operation history. Simple prefix matching would reuse incorrect states. AKV extends SGLang’s radix tree with drop and reposition nodes, freeing dropped KV only when no active request still needs it. Segment-batch attention restores evicted states in one prefill pass.

Across serving benchmarks on 80 long-context SWE-bench Verified tasks, AKV delivered 1.18x output throughput and 1.53x input throughput compared to standard methods. P95 request latency improved by 1.51x. At concurrency 16, P95 latency decreased from 229.90 seconds for standard serving to 30.63 seconds with AKV, reducing per-step full-attention time from 7.082 milliseconds to 2.849 milliseconds.

What to test

Before deploying AKV in production, teams should validate several key claims:

• Latency at scale. The reported 1.18x output throughput and 1.53x input throughput gains come from peak settings. Reproduce these numbers with your own inference cluster and workload patterns, particularly at high concurrency levels.

• Accuracy on your specific tasks. The 1.7-point average improvement derives from a particular benchmark mix. Test AKV against your agent’s actual task distribution. On BrowseComp-Plus with MiniMax-M2.7, the system scored 58.9, which falls below Text Engineering’s 62.3.

• Policy tuning requirements. AKV ships with built-in turn-based, token-based, and model-based policies. The research showed MiniMax-M2.7 reduced cost 17.8% by tuning policies on a small training set. Measure the engineering effort required to find effective policies for your specific use case.

• Model and framework support. Currently AKV supports only Qwen, GPT-OSS, and MiniMax-M2 text models. Confirm your target models are included. The system requires SGLang and runs on NVIDIA GPUs using tensor parallelism.

• Cache hit billing behavior. Cost estimates assume repositioned tokens are charged at the cache-hit rate, a deliberately conservative assumption. Verify actual billing behavior with your inference provider to confirm expected savings materialize.

The conclusion

AKV represents a shift in how serving infrastructure can help agents manage compute costs. By separating message text from KV cache visibility, it creates flexibility for agents to maintain richer context without proportional cost increases. The 27.7% cost reduction and 1.7-point accuracy improvement demonstrate the tradeoff is valid.

The system is not universally applicable across all inference workloads. It requires explicit cache policy design and works best on agentic tasks with repeated tool interactions. Teams running simple completion requests likely will not benefit. However, for agent workloads with long-horizon reasoning tasks, AKV cache engineering merits careful evaluation. Future development should expand model support beyond the current three families and evaluate integration with additional serving frameworks beyond SGLang.

Related reading

• MiniMax M3.1 Flash: $0.15/M tokens cuts agent costs 70%
• Agensh: Decentralized multi-agent harness for 1,024 agents
• Anthropic OSS Scanner free vulnerability detection for open source