Kimi K3’s checkpoint is roughly 1.56 TB on disk: 93 layers, 896 routed experts per MoE layer, about 104B activated parameters per token, and a native context window near 1M tokens.
That definitely reads like cluster hardware, yet kimi-k3-in-c runs K3 inference on a single CPU at a measured 8.24 GB peak RSS.
The engine is portable C99, compiles to about 176 KB, and streams the model from storage instead of pretending the checkpoint belongs in memory.
That’s why this is very important to understand:
Model size and inference working-set size are different numbers. For sparse models they differ by three orders of magnitude.
We have spent two years treating deployment as a VRAM budgeting exercise: how much do we need, how hard can we quantize, how many GPUs to shard across, which server coordinates them.
This project asks the systems question instead: what actually has to be resident to produce the next token?
Studying the architecture is still worth your time, because the same reasoning is about to surface in local agents, edge inference, and every serving stack that has to handle sparse models.
Let’s dive deeper.
The architecture is the whole opening
K3 is a native multimodal, agentic Mixture-of-Experts model.
Total parameters ~2.8T Activated per token ~104B Layers 93 KDA layers 69 Gated MLA layers 24 Routed experts 896 per MoE layer Experts selected 16 per token Shared experts 2 Hidden dimension 7,168 Context length 1,048,576 tokens Expert weights native MXFP4
K3’s Stable LatentMoE router selects a tiny subset of the expert pool per token, and roughly 104B parameters in play for each one.
Still enormous, but structurally different from dense: a dense 2.78T model needs every weight matrix available on every forward pass, while a sparse one needs every expert addressable and only the selected experts readable.
Addressable versus resident, and that gap is the entire exploit.
Editor’s note: If you want to dive deep into Local LLMs and Agentic Stack, you can join our Agent Foundry program for hands-on, in-depth trainings.
8.24 GB is a working-set result
The engine splits the checkpoint into memory classes.
About 1.45 TB of it is the routed expert bank, which never has to be permanently resident, so those weights stay on disk.
When the router picks experts for a layer, the runtime fetches only those, multiplies directly against their packed MXFP4 representation, and moves on.
The remainder is the dense trunk, i.e., attention projections, shared experts, router state, norms, roughly 108.8 GB in the packed streaming layout.
Still too big for a laptop, so that gets streamed too, with as many layers pinned as RAM allows.
1.56 TB checkpoint
├── ~1.45 TB routed experts -> stay on NVMe, fetched on demand
├── ~108.8 GB dense trunk -> pin what RAM allows, stream the rest
└── small resident state -> embeddings, recurrent state, scratchRAM becomes a dial instead of a fit-or-fail condition.
More memory pins more trunk and retains more experts; less memory reads more bytes per token.Same weights and same greedy output but different latency.
That is a more interesting property and it is the one you would deliberately design for.
Inference engines for sparse models look like storage engines
Take one routed layer.
Score, select, fetch, accumulate:
scores = route(hidden_state);
selected = top_k(scores, 16);
prefetch(selected);
for (e in selected) {
w = load_expert_from_cache_or_disk(e);
partial = expert_forward(hidden_state, w);
output += routing_weight(e) * partial;
}Each packed expert is about 17.5 MB, and one token touches routed experts across 92 MoE layers, up to 1,472 expert reads per token.
Serial random reads would be fatal, so the runtime batches the top-k requests, sorts them into an order storage actually likes, issues them in parallel, and holds results in a bounded cache.
If you have built a database, this should be familiar.
- Routing is a query planner
- RAM is a buffer pool
- NVMe is the backing store
- Layer order is a predictable access schedule
- Prefetch is speculative I/O
- Model architecture decides your cache locality
The expert cache barely helps
The obvious move with a huge expert bank and a small hot set is an LRU.
The repo built one, measured it, and found it contributed almost nothing at small memory budgets.
K3 uses Quantile Balancing to flatten expert utilization, which is good for training because no expert gets overloaded, and hostile to a small cache because there is no dominant hot subset to retain.
Spending RAM on the dense trunk beat spending the same RAM on the expert cache until the trunk was nearly pinned.
At a fixed 128 GB budget, shifting the split between trunk and cache produced a reported ~1.69x difference in one sweep.
So bytes moved on the expensive path become objective function.
A cache with an excellent hit rate still loses if it protects the smaller source of I/O while the runtime re-streams a much larger deterministic region every token.
Anyone tuning KV caches, prompt-prefix caches, retrieval caches, tool-result caches, or embedding caches in an agent stack is one measurement away from the same mistake.
Quantized weights as a compute format
K3’s routed experts ship in MXFP4, and the engine’s matmul kernel consumes that packed layout directly instead of expanding a selected expert to FP32 first.
A packed expert is about 17.5 MB and the widened version is a multiple of that.
Multiply the inflation across hundreds or thousands of expert reads per token and you have manufactured a memory-bandwidth problem to replace the storage problem you just solved.
Treating low precision as a first-class compute format rather than a compressed file format becomes table stakes as more checkpoints ship natively quantized.
Bounded state changes the economics of long sessions
K3 is sparse in attention too: 69 Kimi Delta Attention layers and 24 gated MLA layers.
The KDA layers carry recurrent state whose size does not grow with sequence length the way a full KV cache does, and the MLA layers are what add per-position cost during incremental decode.
The C implementation fixed recurrent state for the KDA portion.
That is directly relevant to agents.
Long-running agents are memory-hostile by construction, e.g. every observation, tool result, plan revision, diff, and terminal dump wants to extend the context.
Architectures that bound part of that growth change what a persistent session costs to keep alive.
One caveat, this engine caps prompts at 32,768 tokens today and chunked prefill is still on the roadmap. K3 supports roughly 1M tokens.
Run the tests before you download 1.56 TB
The best decision in this repository is that you can validate the implementation without the weights.
On Linux x86-64 with an AVX2/FMA-capable CPU:
git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j
make testThe suite runs committed fixtures against a small model with the same tensor graph, covering kernels, routing, greedy and incremental decoding, tokenizer behavior, streaming and cache logic, and reference parity.
Prove the engine works on your machine before you spend a day on downloads.
Then inspect the memory presets with ./bin/k3 --list-presets:
preset trunk expert cache peak RSS laptop 3 GB 1 GB ~8.2 GB desktop 16 GB 10 GB ~31.9 GB workstation 60 GB 30 GB ~95.5 GB server 110 GB 13 GB ~128 GB max 110 GB 109 GB ~224 GB
The published numbers do not put max ahead of server.
Consistent with the cache finding above, the last 96 GB of expert cache buys less than it costs.
Full setup
If the tests pass, storage is your first real constraint: about 1.56 TB for the checkpoint, 109 GB for the packed trunk, roughly 1.7 TB free in total, on local NVMe if you want any of this to be tolerable.
Plus Linux x86-64, AVX2 and FMA, GCC 9+ or Clang 10+, and Python 3.9+ for the download and packing utilities.
./scripts/k3-doctor.sh # toolchain, RAM, storage; recommends a preset
export HF_TOKEN=your_read_token # Hugging Face read token
./scripts/download-model.sh ~/k3model # resumable, verifies shard sizes
./scripts/pack-trunk.sh ~/k3model ~/k3trunk # dense layers at known offsets
Packing places each dense layer at a known offset so the runtime fetches a layer with one predictable read instead of reassembling it from scattered checkpoint fragments.
./bin/k3 ~/k3model \
--trunk ~/k3trunk \
--preset laptop \
--tok ~/k3model \
--prompt "The capital of France is" \
--gen 8 \
--incrementalKeep --incremental.
It carries recurrent and KV state forward instead of recomputing the prefix for every generated token.
Without it, every token pays for the whole prefix again and generation gets progressively more expensive.
Make correctness reproducible before chasing speed
There are three input modes: --prompt TEXT, --prompt-file PATH, and --ids 1,2,3.
- For non-ASCII text use a file so the runtime consumes the exact bytes you intended.
- For regression work, feed token IDs directly and dump structured output:
./bin/k3 ~/k3model \
--trunk ~/k3trunk \
--preset desktop \
--ids 1008,10484,318,15383,387 \
--gen 8 \
--incremental \
--out run.jsonThat removes the tokenizer as a variable and gives you a clean diff across builds, memory budgets, and kernel changes.
Small API and no hidden machinery
The public surface is include/k3/k3.h and include/k3/k3_cfg.h.
Configuration, scratch memory, weight location and expert loading are explicit:
#include <k3/k3.h>
#include <k3/k3_cfg.h>
K3Cfg cfg;
int attention_layers[128];
if (!k3_cfg_load_file(&cfg, attention_layers, 128, "model/config.json"))
return 1;
size_t scratch_count = k3_layer_scratch(&cfg, token_count);
float *scratch = malloc(scratch_count * sizeof(float));
for (int layer = 0; layer < cfg.n_layers; ++layer)
run_layer(layer, &cfg, scratch); /* bind or stream, then execute */This is the inverse of a modern Python serving stack, where device placement, graph lowering, kernel selection, allocation, and caching happen behind abstractions you did not choose.
You should read this one end-to-end even if you never touch the checkpoint.
Correctness > Performance
The repo validates in layers: Kernel tests, tokenizer parity, tiny-model end-to-end reference checks, teacher forcing, greedy decode, incremental decode, layer-by-layer comparison against the released checkpoint, and elementwise logit comparison against an independent PyTorch reference.
This is very important because the inference ecosystem is full of optimizations that are faster because they quietly changed the computation, and agents are the worst possible place for that.
A small logit difference flips a tool call, which changes an observation, which changes the next prompt, and two trajectories that started identical end up nowhere near each other.
When you swap a runtime, quantization scheme, prompt cache, model router, speculative decoder, or context-compaction strategy, the first question is whether behavior changed.
Performance reality
Now the uncomfortable part.
At the smallest tier the repo measures roughly 26.5 seconds per token on its test machine.
With enough RAM for the dense working set to sit effectively resident, the README reports around 5.6 seconds per token on the same hardware.
Neither number belongs in an interactive loop.
An agent step generates hundreds of tokens, calls a tool, reads the result, and generates again; at 20+ seconds per token a single planning step runs for hours.
So no, this does not replace a hosted coding-agent endpoint with an 8 GB CPU box.
What it removes is capacity as a hard gate.
On commodity hardware you can now inspect full-scale behavior, validate architecture changes, exercise tokenizer and routing logic, build tooling against the real checkpoint format, study expert locality, profile I/O patterns, test cache policies, and run correctness checks against a frontier-scale model without owning a GPU cluster.
That is a genuine expansion of who gets to work on frontier inference systems.
At low memory budgets, storage is the accelerator
The least glamorous measurement is the most operationally relevant: disk behavior dominates.
The runtime uses direct I/O and large expert reads, so ordinary sequential benchmarks tell you very little, and the repo ships tooling that measures the device with an access pattern closer to the engine’s.
Local NVMe, a consumer SSD, and network-attached storage produce wildly different token latency on identical CPU and RAM.
GPU-centric: weights -> VRAM -> compute
sparse streamed: NVMe -> bounded RAM working set -> CPU computeThe second path is hopeless when every byte is needed for every token, and plausible the moment sparsity makes the working set small relative to the checkpoint.
The likely destination is several tiers at once, i.e., HBM, system RAM, local NVMe, remote object storage, with the runtime deciding what belongs where and when to move it.
What this means if you build agents
Inference architecture is starting to constrain product design directly, in five specific ways.
1. Sparse checkpoints create new deployment surfaces
A checkpoint can be enormous while its per-token working set stays small, which favors runtimes optimized for weight availability over weight residency. For private or offline agents, that may eventually matter more than whether a model fits inside one GPU.
2. Latency amplification is an agent problem
Disk-streamed inference is for research, low-throughput batch work, validation, and architecture exploration today.
3. Working-set observability deserves to be a first-class metric
Parameter count is a useless operational number for an MoE model. Instrument active parameters per token, bytes read per token, expert reuse distribution, dense trunk residency, KV and recurrent state growth, storage queue depth, cache retention across turns, and prefill versus decode cost.
4. Every optimization needs a correctness gate
Errors compound across steps, so agent systems are unusually sensitive to small inference changes. Pin behavior with deterministic parity checks first, then tune, then re-run the checks.
5. Serving is systems engineering again
The wins in this project are memory classification, file layout, direct I/O, prefetching, cache policy, quantized kernels, recurrent state, and failure semantics. Agentic AI is pulling application developers back toward compilers, databases, operating systems, and distributed systems. Staff accordingly.
What it does not do yet
Linux x86-64 only, AVX2 and FMA required, CPU only with GPU support explicitly out of scope, greedy decoding only, no chat template so you get base-model continuations rather than an assistant protocol, no vision pipeline despite K3 being natively multimodal, no HTTP serving API, prompts capped at 32,768 tokens, chunked prefill unimplemented, and one inference at a time per cache and trunk instance rather than concurrent serving.
Concluding Thoughts
A 2.78T model on an 8 GB CPU machine breaks an assumption that has quietly shaped the entire local-AI stack: that the full checkpoint has to live where the compute happens.
Once that goes, RAM, NVMe, routing, quantization, and recurrent state stop being separate concerns and become one design space.
Keep the working set hot, keep everything else addressable, and move only the bytes the next operation needs.
That is how databases have treated terabytes for decades. It is a bigger idea than the 8.24 GB headline!