VRAM is no longer the hard upper bound on the model you can serve interactively.
Until very recently, you only had two compromises.
Either you ran a small model that fit comfortably on a consumer GPU, or you aggressively quantized a much larger model until it technically fit.
That assumption is starting to break.
There is a new edge-native Mixture-of-Experts serving engine that dynamically orchestrates GPU VRAM, CPU compute, system memory, and PCIe bandwidth.
The published numbers are difficult to ignore:
That means a laptop GPU can serve a 35B MoE model at interactive speed, while a gaming desktop can serve a model in the 284B class. You can point it to an existing coding-agent workflow at a local frontier-scale model and remove the per-token API bill.
You still own the hardware, pay for power, and need enough host RAM and storage for the checkpoint but the marginal API cost becomes effectively $0.
The Problem Was Never Just “Does the Model Fit in VRAM?”
FreeToken exploits the structure of modern Mixture-of-Experts models.
A model such as Qwen3.6–35B-A3B has 35B total parameters but activates roughly 3B for a token.
DeepSeek-V4-Flash is hundreds of billions of parameters in total, but only a fraction of the experts participate in each decoding step.
That sparse activation makes the computation feasible but the problem is memory movement.
The full expert pool still exists.
If it does not fit in VRAM, the runtime needs to decide what stays on the GPU, what stays in host memory, what crosses PCIe, and what can execute directly on the CPU.
Traditional local inference engines often make that decision statically.
Put some layers or experts on the GPU, leave the rest on the CPU, then live with the placement, which is a poor match for agentic workloads.
- During decode, expert routing changes token by token.
- During prefill, a long prompt can touch almost every expert across the model.
- During an agent session, tool calls and context edits repeatedly force the serving engine back into prefill.
FreeToken treats all three as scheduling problems rather than fixed model-placement problems.
Editor’s note: If you want to dive deep into Local LLMs and Agentic Stack, you can join our Agent Foundry program for hands-on, in-depth trainings.
Why FreeToken Is Faster
It’s roughly 3–4x faster decode and 6–30× faster prefill than Ollama in selected configurations.
On an RTX 5090, FreeToken sustains 77–83 tok/s on Qwen3.6–35B-A3B and 22–25 tok/s on DeepSeek-V4-Flash across real agentic workloads.
Against the strongest available edge-serving baseline in those tests, decode improves by roughly 1.5–2.3x.
The speedup comes from several pieces working together.
Bandwidth-adaptive CPU–GPU execution
When a routed expert misses the GPU cache, the obvious solution is to copy it over PCIe but that is not always optimal.
The CPU can also execute an expert directly from host memory.
- If you send every miss to the GPU, CPU memory bandwidth and cores may sit underused.
- If you execute every miss on CPU, you leave PCIe and GPU compute unused.
FreeToken measures the machine and chooses the split dynamically.
The paper describes this as the q* policy: divide missing experts between GPU cache fills and direct CPU execution based on measured PCIe bandwidth and host-memory bandwidth.
The important idea is that the optimal offload policy is a property of the actual machine, not the model alone.
A laptop with an RTX 4060 over a narrower PCIe link should not use the same policy as a desktop RTX 5090 with PCIe 5.0 and different DDR5 bandwidth.
FreeToken can calibrate this directly:
ft bench bw
That writes a bandwidth profile that the runtime can use when selecting the hybrid MoE backend.
Prefill is pipelined instead of serialized
Prefill is especially painful for large MoE models.
A single decode token only routes through a handful of experts. A prompt containing thousands of tokens tends to collectively activate most of them.
So prefill behaves much more like a dense workload: a large portion of the expert pool has to move through the system.
FreeToken uses full-layer double buffering.
While the GPU computes layer l, the experts for layer l+1 are streamed over PCIe into the second buffer.
Qwen3.6 prefill throughput reaching about 6.7K tok/s at a 16K-token prompt on an RTX 5090, with expert computation effectively hidden behind transfer.
Semantic-aware caching is designed for agent turns
Agent harnesses do not always append tokens to an immutable conversation, they edit context.
Old tool outputs may be pruned, thinking blocks may be removed, compaction may replace parts of the conversation and a new tool result can change the effective prompt prefix.
For hybrid-attention or recurrent models, those edits can invalidate cached state and force thousands of tokens to be recomputed.
FreeToken places recurrent-state checkpoints at semantic boundaries such as thinking segments and tool calls.
When the context changes, it can restore the deepest checkpoint that still matches and recompute only the new suffix.
In other words, the cache is aware that an agent conversation has structure.
That is much closer to how modern coding agents actually behave than treating every request as an unrelated prompt.
Quick Start: Run Qwen3.6 Locally
FreeToken has a desktop app for Windows and Linux.
The easiest path is to install the app, pick a model, pick an agent, and run.
For developers who want the server directly, the current CLI documentation targets Linux x86_64 with an NVIDIA GPU, driver r580+, CUDA 13, and Python 3.10 or newer.
Install with uv:
uv venv
source .venv/bin/activate
uv pip install "freetoken[accel]"The CUDA kernels are JIT-compiled on first use, so the documented CLI path expects a CUDA 13 toolkit with nvcc available.
Before serving a large MoE model, benchmark the machine once:
ft bench bw
Then start a server.
FreeToken accepts either a local checkpoint path or a Hugging Face repository ID.
For example, the repository lists NVIDIA’s official NVFP4 Qwen3.6 checkpoint as a known-good model:
ft serve --model nvidia/Qwen3.6-35B-A3B-NVFP4
The server is ready when it reports that the API is listening on:
127.0.0.1:1919
Check the served model:
curl http://127.0.0.1:1919/v1/modelsThen send a normal OpenAI-style chat request:
curl http://127.0.0.1:1919/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen3.6-35B-A3B-NVFP4",
"messages": [
{
"role": "user",
"content": "Explain why this repository has a circular dependency."
}
],
"max_tokens": 512,
"stream": true
}'FreeToken exposes /v1/chat/completions, /v1/responses, and /v1/models, plus Anthropic-compatible /v1/messages and token-counting endpoints.
That means you do not need a custom SDK to integrate it into an existing agent platform.
Point Claude Code or Codex at the Local Model
This is where FreeToken becomes more interesting than a local chat server.
The repository includes an ft launch command that discovers the model exposed by the server, writes the target agent’s provider configuration, installs the CLI if necessary, and launches the application against FreeToken.
Preview what it will change:
ft launch claude --dry-run
Then launch Claude Code against the local server:
ft launch claude
Or Codex:
ft launch codex
The same command supports other harnesses:
ft launch opencode ft launch openclaw ft launch dsh ft launch hermes
One subtle but important implementation detail: the launcher clears cloud API keys such as ANTHROPIC_API_KEY and OPENAI_API_KEY from the child process.
That prevents an agent from silently falling back to a paid cloud provider because a credential happened to exist in your shell.
Use It as a Backend for Your Own Agent
Because the API is compatible with common chat-completion formats, an internal agent service can treat FreeToken like any other inference endpoint.
A minimal Python request looks like this:
import requests
response = requests.post(
"http://127.0.0.1:1919/v1/chat/completions",
json={
"model": "Qwen3.6-35B-A3B-NVFP4",
"messages": [
{"role": "system", "content": "You are a repository maintenance agent."},
{"role": "user", "content": "Find the likely cause of the failing tests."}
],
"max_tokens": 800
},
timeout=120,
)
print(response.json()["choices"][0]["message"]["content"])This makes the migration path straightforward.
Your application does not need to understand expert caches, PCIe scheduling, recurrent-state checkpoints, or host-memory bandwidth.
It means you can benchmark FreeToken behind the same evaluation harness you already use for hosted APIs instead of rewriting the agent around a local-runtime abstraction.
No GGUF Conversion Is a Bigger Deal Than It Sounds
Local LLM workflows often accumulate a surprising amount of model-format plumbing.
Download a checkpoint, find the right community quantization, convert to GGUF, pick a quantization level, discover that tool calling behaves differently and re-download another variant.
FreeToken loads Hugging Face safetensors checkpoints directly for its supported model families.
Its model list includes Qwen3.6, DeepSeek-V4, GLM-5.2, GLM-4.7, GPT-OSS, MiniMax, Gemma, and others.
There is an optional FreeToken Weight format, FTW, for faster loading:
ft checkpoint \
--model <hf_dir> \
--out <ftw_dir> \
--moe-backend offloadBut conversion is optional.
You can start from official checkpoints, keep model provenance clear, and avoid making an unofficial GGUF build a hidden dependency in your production-ish local stack.
Also, no extreme quantization does not mean everything runs in BF16.
RTX 4060 Qwen3.6 result uses NVFP4, DeepSeek-V4-Flash uses MXFP4 in the RTX 5090 experiments, and the GLM-5.2 workstation result uses NVFP4.
The point is that these are supported, high-quality checkpoint formats rather than an ad-hoc squeeze-to-fit exercise.
What I Would Monitor in Developer Setup
FreeToken exposes operational commands instead of forcing you to infer everything from nvidia-smi.
ft ctl health ft ctl stats ft ctl cache
The CLI can also resize cache pools on a running server without restarting the engine.
That is useful because agent sessions change memory pressure over time: the expert cache wants VRAM, but so does an expanding KV cache.
On a workstation, I would treat FreeToken as a small internal inference service rather than a desktop toy: run one known checkpoint, expose the API only on the trusted network you intend, collect latency and cache metrics, and keep the agent harness separate from the model server.
The large-model tiers also need realistic host resources.
A 753B model has not somehow become an 8GB model.
FreeToken keeps the complete expert pool in the machine’s memory hierarchy and uses the GPU as the high-speed execution/cache tier.
Fast NVMe, substantial system RAM, and PCIe topology still matter.
The breakthrough is that VRAM is no longer the hard upper bound on the model you can serve interactively.