Kimi K3 Changed the Memory Math for Agentic AI Forever

Disclosure: This article discusses the possible effect of model architecture on infrastructure and memory demand. It is not investment advice.

Why KV Cache Became an Agent Infrastructure Problem

During autoregressive decoding, a Transformer reuses the keys and values computed for previous tokens.

Recomputing them for the entire context on every output token would be wasteful, so inference engines store them in a KV cache.

For a simplified full-attention layer, cache memory scales like:

KV bytes ≈ tokens × layers × cached dimensions × bytes per value

The exact number depends on the attention architecture, precision, batching, parallelism, paging, and implementation.

The variable that matters is tokens, and it only grows.

That growth was manageable when an application sent a few thousand tokens and generated a short response.

It becomes a platform concern when an agent:

• loads a large repository
• keeps a persistent session for hours or days
• calls dozens or hundreds of tools
• receives screenshots, logs, traces, and documents
• explores several branches before choosing a solution;
• maintains multiple concurrent workspaces
• reuses a large system prompt and tool catalog

In those workloads, context length becomes a memory reservation.

That reservation shapes admission control, batching, cache eviction, prefill latency, prefix reuse, session migration, failure recovery, and how many concurrent agents a cluster can hold.

A million-token context is not useful if one active session crowds out the rest of your users.

Linear Attention’s Promise

Linear attention replaces the growing token-by-token cache with a recurrent state.

A deliberately simplified mental model:

The state has a fixed shape.

Whether the model has processed a thousand tokens or a million, it keeps updating the same tensor, which makes storage for those layers sequence-independent rather than sequence-dependent.

The catch is that this is a compression system.

A fixed state has to summarize an unbounded stream, so some information survives intact, some is blended, and some is discarded.

• Full attention behaves more like retaining an addressable record for every token.
• Recurrent linear attention behaves more like maintaining a learned, continuously updated data structure.

The failure mode is easy to picture.

Ask the model for a specific variable name, checksum, migration ID, or error string from much earlier in a long session, and a compressed state may retain the gist of the discussion while losing the exact tokens.

That is why pure linear-attention models have historically struggled to replace full attention at the frontier.

What Kimi Delta Attention Adds

Kimi Delta Attention, or KDA, extends Gated DeltaNet with more fine-grained control over memory decay.

The useful intuition is selective forgetting.

Earlier recurrent designs applied a relatively coarse decay to a head’s memory, while KDA uses channel-wise gating so different dimensions of the state can decay at different rates.

Conceptually:

Here, D_t represents learned, per-channel decay rather than one uniform forget value for the entire state.

This is not the full KDA equation, but it captures the engineering intuition: the update policy can retain some memory channels while aggressively recycling others.

K3’s published configuration makes the state concrete:

• 69 KDA layers;
• 96 attention heads;
• key dimension 128;
• value dimension 128;
• BF16 model dtype.

The KDA state for one session is therefore approximately:

69 layers × 96 heads × 128 × 128 × 2 bytes = 217,055,232 bytes ≈ 0.22 GB

That state is effectively independent of sequence length.

KDA also needs convolution state and serving metadata, and real engines add allocation overhead but the dominant recurrent matrix never appends an entry per token.

Moonshot has open-sourced FlashKDA, a CUTLASS-based implementation of the KDA forward kernel.

The repository exposes the state as a tensor shaped like [batch, heads, value_dim, key_dim], which makes the constant-state property visible in code rather than only in a diagram.

Editor’s note: If you want to dive deep into Local LLMs and Agentic Stack, you can join our Agent Foundry program for hands-on, in-depth trainings.

Why the Hybrid Is More Important Than the Linear Part

K3 is not a pure linear model.

It interleaves two layer types in a roughly 3:1 pattern:

• KDA, 69 layers: fixed recurrent state, carrying efficient long-range updates.
• Gated MLA, 24 layers: KV cache that grows with tokens, providing high-fidelity global retrieval.
• Together, 93 layers: a hybrid that buys efficiency without giving up exact token access.

A pure recurrent stack asks finite-state memory to do everything, K3 instead runs most sequence processing through KDA and periodically routes information through full-attention layers.

You can think of it as a database with two storage tiers:

• KDA is the compact, continuously maintained index;
• MLA is the smaller set of addressable records that remains available for exact global interaction.

That analogy is imperfect, but it leads to the right operational conclusion: K3 saves substantial memory without claiming that lossy compression alone can preserve every detail.

Moonshot’s earlier Kimi Linear experiments reported that the hybrid architecture could match or outperform an all-MLA baseline while reducing KV-cache use by up to 75% and improving million-token decoding throughput.

K3 is the frontier-scale validation of that direction.

The Memory Math at 128K and 1M Tokens

Let us make the claim testable.

The public K3 configuration specifies:

For simplified MLA cache accounting, use 512 + 64 = 576 BF16 values per token per layer. This represents the compressed KV latent plus the RoPE component.

Baseline: all 93 layers using MLA

Actual K3: 24 MLA layers plus 69 KDA layers

In decimal gigabytes, using only these two formulas:

• At 128,000 tokens: 13.71 GB for the hypothetical 93-layer MLA stack versus 3.76 GB for K3, a reduction of about 72.6%.
• At 1,000,000 tokens: 107.14 GB versus 27.87 GB, a reduction of about 74.0%.

The KDA state is roughly 0.22 GB in both cases.

Almost all of the growth on the hybrid side comes from those 24 MLA layers.

“But KDA State Is Not Additive”

A valid systems objection is that a KV cache can be paged and shared at token-block boundaries, while recurrent state depends on the entire prefix that produced it.

Suppose an agent processes one million tokens and you want restartable checkpoints every 20,000 tokens.

That is 50 state snapshots, at roughly 0.22 GB each:

50 snapshots × 0.217 GB ≈ 10.85 GB

Add that to the active hybrid cache:

27.87 GB active state/cache + 10.85 GB snapshots ≈ 38.72 GB

That is still about 64% below the simplified 107.14 GB all-MLA comparison.

It means the recurrent-state objection changes the storage model but it does not erase the architectural saving.

The vLLM K3 implementation had to redesign hybrid prefix caching for exactly this reason.

Full-attention layers use paged KV blocks, while KDA layers need recurrent-state snapshots. vLLM decouples the physical KDA state-block size from fine-grained prefix matching so shared prompts can reuse both forms of memory.

K3 forced a major inference engine to treat recurrent-state caching as a first-class serving primitive.

What About Subagents?

Agent systems often use a coordinator that launches specialized workers.

A common synchronous pattern:

The subagent’s cache exists while it runs, but it does not necessarily become a permanent addition to the coordinator’s footprint.

The long-lived state is usually the coordinator session itself.

The harder case is asynchronous work:

Long-running concurrent subagents do multiply state requirements, because KDA cannot repeal concurrency.

The practical design rules are straightforward:

• keep short-lived exploration workers ephemeral
• return compact artifacts to the coordinator
• persist checkpoints for recoverability, not every transient step
• set budgets for concurrent long-lived branches
• measure active state per workflow, not only per request.

Hybrid attention reduces the cost of each long-running branch. It does not make unlimited branching free.

Scaling Laws Are Engineering Curves

Scaling laws are observations about a model family, data regime, and training process.

They are not physical constants, change the architecture and the curve can move.

K3 combines:

• a better finite-state attention mechanism
• periodic full attention
• selective residual routing across depth
• more efficient expert computation
• low-precision weights
• custom kernels
• long-context training and agentic reinforcement learning
• serving-engine changes for hybrid cache management

The result is something more useful:

Frontier capability does not require accepting the memory slope of yesterday’s architecture.

That is progress.

Concluding Thoughts

Kimi K3’s most consequential contribution is evidence that hybrid linear attention can survive frontier-scale training and deliver strong long-horizon behavior while materially reducing the cache growth that makes persistent agents expensive.

You should also update your assumptions:

1. KV cache growth is an architectural choice.
2. “Constant state” applies to KDA layers.
3. Hybrid designs can preserve full-attention retrieval while reducing the memory slope by roughly three quarters in a simplified comparison.
4. Recurrent state creates new checkpointing, prefix-cache, and migration problems.
5. Agent platforms should manage context like a storage system.
6. The best first experiment is API-level workload testing, not self-hosting 2.8T parameters.
7. The architecture to watch is hybrid memory with explicit serving support.

The next generation of agents will become practical because the systems underneath them learn to remember more efficiently.