744B MoE on 25 GB of CPU RAM!!!

744-billion-parameter model should not run on a 25 GB consumer machine.

At least, not according to the deployment architecture most of us use today.

The normal assumption is simple: model weights belong in GPU memory or system RAM.

If the model does not fit, quantize it harder, shard it across accelerators, buy more hardware, or use an API.

Colibrì takes a different path.

It keeps the dense, always-needed part of GLM-5.2 resident in RAM and treats the model’s routed experts as a disk-backed working set.

The experts stay on NVMe and the router decides which experts are needed, Colibrì streams only those experts into memory as inference progresses.

The result is strange enough to sound fake: GLM-5.2, a roughly 744B/753B-parameter Mixture-of-Experts model, running in pure C on a machine with around 25 GB of RAM.

No GPU is required.

The engine is centered around a roughly 2,400-line C file.

The trade-off is equally important: this is not fast inference on low-end hardware.

Colibrì’s own baseline reports roughly 0.05–0.1 tokens/second on a 12-core WSL2 machine with 25 GB RAM and an NVMe path limited to around 1 GB/s random reads.

But the interesting part is the systems design rather than speed.

For developers building local coding agents, private agentic workflows, long-running research agents, or experimental AI infrastructure, Colibrì is a useful example of what happens when you ask “What does one token actually need right now?”

Let’s dive into details.

Editor’s note: To celebrate reaching 10,000 community members on Medium, who relentlessly design, ship, and iterate on agents every day, we’re also making the full repository available for free, which is part of our Agent Foundry program.

The key idea: model size is not the same as per-token working set

GLM-5.2 is a Mixture-of-Experts model.

It’s Z.ai’s flagship model for long-horizon tasks, with a 1M-token context, stronger coding capability, multiple reasoning-effort levels, sparse-attention improvements, and an improved multi-token prediction layer for speculative decoding.

A dense 744B model would make Colibrì’s approach much harder but MoE changes the shape of the problem.

GLM-5.2 activates only around 40B parameters per token.

More importantly for the engine’s architecture, only around 11 GB of routed-expert weights change from token to token.

The engine splits the model into two broad classes of data:

• Model component Colibrì strategy
• Attention, shared experts, keep resident in RAM embeddings, other dense weights

21,504 routed experts keep on disk and stream on demand, compressed KV state keep in memory and optionally persist across sessions

Frequently used experts promote into RAM or optional VRAM hot tiers.

Cold experts Read from local NVMe when routed

The dense resident set is about 9.9 GB at int4 and the routed experts occupy roughly 370 GB on disk.

This is the architecture in one sentence:

RAM is the model’s stable working set. NVMe is the expert store. The router is effectively part of the I/O scheduler.

In a normal LLM runtime, routing is discussed as a compute concern.

Which expert processes this token?

In Colibrì, routing also creates a storage access pattern.

Every layer can trigger reads for expert weights, a cold token may require roughly 11 GB of disk reads across 75 MoE layers and eight routed experts per layer.

So the problem changes from “execute a huge neural network” to a combined inference, caching, prefetching, and storage-scheduling problem.

Why this matters for agentic AI

We spend a lot of time on model routing:

• use the cheap model for classification;
• use the coding model for repository changes;
• escalate hard tasks to a frontier model;
• cache prompts;
• reduce context;
• add a smaller verifier.

All valid.

But there is another architectural option: change how the model itself is materialized at inference time, agentic workloads contain natural latency-hiding opportunities.

The agent can perform deterministic work outside the model.

It can parse files, query databases, run builds, execute tests, inspect logs, or wait on remote services.

In these workflows, model latency is one stage in a much larger pipeline.

Colibrì is not currently a tool-calling agent runtime, its OpenAI-compatible server explicitly rejects tools and functions.

That limitation should be stated clearly but it can already act as a text-generation and reasoning backend behind an agent orchestrator.

The orchestration layer owns tool execution and Colibrì owns model inference.

For a first integration, that separation is more than enough.

Colibrì’s architecture is deliberately boring

The repository layout is refreshingly small:

Makefile c/ ├── glm.c ├── st.h ├── tok.h ├── json.h ├── backend_cuda.* ├── Makefile ├── coli ├── openai_server.py ├── setup.sh ├── tools/ ├── scripts/ └── tests/ web/

glm.c contains the GLM engine, small headers handle safetensors, tokenization, and JSON.

coli is the user-facing CLI, openai_server.py is the OpenAI-compatible gateway, and python exists for conversion and the HTTP wrapper, but the actual inference engine is C.

The engine is pure C with zero runtime dependencies.

The one-time FP8-to-int4 conversion uses Python packages including torch, safetensors, huggingface_hub, and numpy, but once the model is converted, inference runs through the C engine.

There is also a pre-converted model container on Hugging Face, GLM-5.2-colibri-int4, so you can skip local FP8 conversion.

The container is not GGUF, AWQ, GPTQ, or MLX but it is Colibrì’s own format: packed int4 weights plus per-row scales aligned with the engine’s C quantization kernels.

Quick start: run GLM-5.2 locally with Colibrì

You need a local NVMe drive, do not put the model on a network mount.

The project currently targets Linux, WSL2, macOS, and native Windows 11 through MinGW-w64.

You should have specs: AVX2, GCC with OpenMP, at least 16 GB RAM, and roughly 370–400 GB of local NVMe storage for the int4 model.

The fastest path is to use the pre-converted model.

1. Clone and build Colibrì

git clone https://github.com/JustVugg/colibricd colibri/c ./setup.sh

2. Download the pre-converted int4 model

Install the Hugging Face CLI if needed, then download the Colibrì container to a fast local disk:

hf download jlnsrk/GLM-5.2-colibri-int4 \
--local-dir /nvme/glm52_i4

The model repository is around 370 GB.

The Hugging Face model card explicitly recommends a fast local NVMe path and warns against network or 9p mounts.

3. Inspect the resource plan before loading 744B parameters

This is one of my favorite features in the repository.

Run:

COLI_MODEL=/nvme/glm52_i4 ./coli plan

For a machine with GPUs and a larger RAM budget:

COLI_MODEL=/nvme/glm52_i4
./coli plan --gpu 0,1 --ram 128 --vram 48 --json

coli plan reads safetensors headers and calculates the dense footprint, runtime reserve, safe expert-cache cap, and bounded VRAM hot tier.

It does not load model tensors or start inference.

That is exactly how heavyweight local AI software should behave.

Resource admission should happen before the process discovers, halfway through initialization, that your memory plan was fantasy.

You can apply the calculated plan to the runtime:

COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier

Explicit environment variables and flags still take precedence.

4. Start a chat session

COLI_MODEL=/nvme/glm52_i4 ./coli chat

The repository’s example looks like this:

$ ./coli chat
colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ pronto in 32s · residente 9.9 GB
› ciao!
◆ Ciao! Come posso aiutarti oggi?

The important number is the 9.9 GB resident dense set, remaining expert capacity is treated as a dynamic storage hierarchy.

The cache is the real product

If Colibrì simply read 11 GB from disk for every token forever, the project would be a technical demo.

Colibrì uses several layers of locality:

1.  a per-layer LRU expert cache;
2.  an optional pinned hot-expert store;
3.  the operating system page cache;
4.  persistent expert-usage statistics;
5.  optional live tier adaptation;
6.  experimental router-lookahead prefetch.

The engine records which experts your workload actually uses in .coli_usage.

At startup, it can use spare RAM to pin the hottest experts, and Colibrì gets faster the more you use it.

Agent workloads are repetitive.

A coding agent working across TypeScript monorepos, Kubernetes manifests, and PostgreSQL schemas does not generate the same routing distribution as a multilingual creative-writing assistant.

If expert usage has workload locality, a persistent heat map can convert repeated disk reads into RAM hits.

On a Ryzen AI 9 HX 370 system with 128 GB RAM, the project reports an expert hit rate reaching 66% with a 46.7 GB auto-learned pin. Another Ryzen AI Max+ 395 result reportedly improved to a 71% hit rate after five runs with a learned 47.6 GB pin.

For agent infrastructure, I would take this one step further.

Use one Colibrì instance per workload class, not one universal instance for the entire company.

For example:

colibri-code
-> coding agents
-> PR review
-> repository analysis
-> test failure diagnosis

colibri-research
-> long-form synthesis
-> document analysis
-> web research planning

colibri-ops
-> incident summaries
-> log interpretation
-> runbook reasoning

Each instance develops a different expert heat map.

I would not mix every workload until I had routing telemetry proving the hot sets overlap.

This is an inference analogue of cache pollution.

Persistent KV changes the economics of long-running agents

Colibrì also persists its compressed KV cache.

The project implements Multi-head Latent Attention with a compressed KV representation, 576 floats per token instead of 32,768, a 57x reduction for GLM-5.2’s attention architecture.

In serve mode and chat mode, Colibrì can append KV state to .coli_kv after every turn, and conversation can resume without re-prefilling the full history.

Long-running agents are often punished twice for context growth.

• First, context consumes memory.
• Second, process restarts force the system to re-tokenize and re-prefill the entire conversation or reconstructed state.

Colibrì’s persisted KV design treats model context as durable runtime state.

Persisted conversations resume with zero re-prefill and were validated byte-identical to uninterrupted sessions.

The rough storage cost is about 182 KB per token and that is not small.

A million tokens would still be enormous.

But the architecture is useful: agent state does not always need to be serialized back into English and replayed through the model.

Sometimes the model’s own inference state is the artifact worth persisting.

There are obvious operational consequences, you now need to version KV state against model weights and runtime format, need crash-safe writes, explicit ownership and lifecycle rules.

Colibrì handles multiple independent contexts through KV slots.

Start the server with:

COLI_MODEL=/nvme/glm52_i4
./coli serve --kv-slots 4

Then select a context with the cache_slot extension:

{
"model": "glm-5.2-colibri",
"messages": [
  {
    "role": "user",
    "content": "Continue this conversation"
  }
],
"cache_slot": 1
}

Each slot owns its token history, compressed KV/DSA memory, MTP window, and persistence file.

The OpenAI-compatible API is intentionally narrow

To use Colibrì behind an application, start the API server:

cd colibri/c

COLI_MODEL=/nvme/glm52_i4 \
COLI_API_KEY=local-secret \
./coli serve \
--host 127.0.0.1 \
--port 8000 \
--model-id glm-5.2-colibri

Then call it with a normal Chat Completions request:

curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Authorization: Bearer local-secret' \
-H 'Content-Type: application/json' \
-d '{
  "model": "glm-5.2-colibri",
  "messages": [
    {
      "role": "user",
      "content": "Review this failure and propose the smallest safe fix."
    }
  ],
  "stream": true
}'

The server implements:

GET  /v1/models
GET  /v1/models/{model}
POST /v1/chat/completions
POST /v1/completions
GET  /health

It supports JSON responses, SSE streaming, usage counts, max_tokens, max_completion_tokens, temperature, and top_p.

You can enable GLM-5.2’s reasoning block with:

{
"model": "glm-5.2-colibri",
"messages": [
  {
    "role": "user",
    "content": "Analyze this distributed transaction failure."
  }
],
"enable_thinking": true
}

The standard reasoning_effort field also enables thinking unless it is set to nonebut this is not a drop-in replacement for every OpenAI client.

The current gateway explicitly rejects unsupported features including tools, functions, image/audio input, custom stop sequences, log probabilities, and token penalties.

The repository code does this intentionally.

From c/openai_server.py:

for name in ("tools", "functions"):
  if body.get(name):
      raise APIError(
          400,
          f"`{name}` is not supported yet.",
          name,
          "unsupported_parameter",
      )

Silently ignoring an unsupported tools field would be disastrous in an agent system. The orchestrator might believe the model has tool access when the backend has discarded the schema.

Fail loudly!

A minimal agent backend integration

Because Colibrì speaks the Chat Completions protocol, you can put it behind a simple orchestration loop.

The model does not call tools natively, so the application owns the action protocol.

A minimal Python example could look like this:

import json
import urllib.request

COLIBRI_URL = "http://127.0.0.1:8000/v1/chat/completions"
API_KEY = "local-secret"

def ask_colibri(messages, cache_slot=0):
  payload = {
      "model": "glm-5.2-colibri",
      "messages": messages,
      "temperature": 0.2,
      "max_completion_tokens": 512,
      "cache_slot": cache_slot,
      "enable_thinking": True,
  }

  request = urllib.request.Request(
      COLIBRI_URL,
      data=json.dumps(payload).encode(),
      headers={
          "Authorization": f"Bearer {API_KEY}",
          "Content-Type": "application/json",
      },
  )

  with urllib.request.urlopen(request) as response:
      return json.load(response)["choices"][0]["message"]["content"]

Now keep tool execution outside the model server:

messages = [ { "role": "system", "content": """ You are a code investigation agent.

Return exactly one action:
READ_FILE <path>
RUN_TEST <command>
FINAL <answer>
""".strip(),
  },
  {
      "role": "user",
      "content": "Find why the payment idempotency test is failing.",
  },
]

while True:
  output = ask_colibri(messages, cache_slot=2)
  messages.append({"role": "assistant", "content": output})

  if output.startswith("READ_FILE "):
      path = output.removeprefix("READ_FILE ").strip()
      result = read_file_from_sandbox(path)
  elif output.startswith("RUN_TEST "):
      command = output.removeprefix("RUN_TEST ").strip()
      result = run_allowlisted_command(command)
  elif output.startswith("FINAL "):
      print(output.removeprefix("FINAL ").strip())
      break
  else:
      result = "Protocol error: return one allowed action."

  messages.append({
      "role": "user",
      "content": f"Tool result:\n{result}",
  })

Use a constrained action grammar, strict validation, allowlisted tools, execution timeouts, output limits, sandboxing, and a maximum-turn budget.

Your agent loop does not need the inference backend to own tools.

• Backend can remain a text model server.
• Application can own policy and execution.

For security-sensitive agent systems, I often prefer this separation anyway.

Colibrì does not pretend concurrency exists where it does not

The engine owns mutable KV state and runs one sequence at a time.

A naive HTTP wrapper could use threads, accept 100 requests, and let them corrupt shared state.

Colibrì instead uses a bounded FIFO admission queue.

The relevant server code is straightforward:

class GenerationScheduler:
  """Bounded FIFO admission for the engine's single mutable KV context."""

  def __init__(self, max_queue=8, queue_timeout=300):
      self.max_queue = max_queue
      self.queue_timeout = queue_timeout
      self.condition = threading.Condition()
      self.queue = collections.deque()
      self.active = False

When the queue is full, the server returns an OpenAI-shaped HTTP 429, and when a request waits too long, it gets a queue-timeout error.

The server exposes queue counters through /health, and successful responses include x-colibri-queue-wait-ms.

MTP speculative decoding is useful, but cache temperature decides whether it wins

GLM-5.2 includes a multi-token-prediction head and Colibrì uses it for native speculative decoding.

The MTP head drafts future tokens, main model verifies the draft in a batched forward pass, and process remains lossless under sampling through rejection sampling.

MTP head also needs int8:

• At int4, draft acceptance collapsing to 0–4%.
• At int8, measurements show 39–59% acceptance and roughly 2.2–2.8 tokens per forward.

This is a useful reminder for anyone quantizing agent models.

Not every model component has the same error tolerance.

Quantizing everything to the same bit width because the deployment config has one **bits=4** flag is not systems engineering because the draft model’s value depends on agreement with the verifier.

Small quantization errors can destroy acceptance rate even when the draft text looks superficially reasonable.

Benchmark your disk before benchmarking the model

Colibrì includes an I/O benchmark that attempts to reproduce the engine’s access pattern: parallel random reads of roughly 19 MB expert blocks.

Run:

cd colibri/c

gcc -O2 -fopenmp iobench.c -o iobench

./iobench \
/nvme/glm52_i4/out-00069.safetensors \
19 64 8 0

./iobench \
/nvme/glm52_i4/out-00069.safetensors \
19 64 8 1

The final argument switches between buffered I/O and O_DIRECT.

Then run a real chat and inspect the per-turn statistics:

COLI_MODEL=/nvme/glm52_i4 ./coli chat

To collect routing frequencies:

STATS=stats.txt ./coli chat

Then explicitly pin hot experts into spare RAM:

PIN=stats.txt \
PIN_GB=20 \
./coli chat

The project now auto-sizes and can auto-raise its expert cache based on available RAM, so manual tuning is less critical than it was in earlier versions.

Still, this workflow teaches you where the bottleneck lives.

The community benchmark table contains a particularly useful hardware comparison.

A Ryzen 9 9950X system reportedly moved the same model from a Crucial P3 Gen3 SSD to a Samsung 9100 PRO PCIe 5.0 drive. I/O bandwidth increased from 1.51 GB/s buffered to 8.81 GB/s with O_DIRECT. Token throughput increased from 0.10 to 0.28 tokens/second.

More interestingly, the runtime profile changed.

The first setup was 66% disk-bound, faster SSD shifted the profile to 57% matmul.

That is classic bottleneck migration because rather than solving the performance, you move the bottleneck.

What I would deploy for a small agent team

I would not replace a low-latency hosted model with Colibrì and route every developer chat request to it.

That would be a bad product decision on slow hardware.

I would use it as a specialized inference tier.

Something like:

Developer / Agent
     |
     v
Agent Orchestrator
     |
     +---- fast hosted model
     |       interactive tasks
     |       short planning
     |       latency-sensitive work
     |
     +---- Colibrì / GLM-5.2
     |       private repository analysis
     |       long-running jobs
     |       overnight maintenance
     |       deep code reasoning
     |
     +---- deterministic workers
             git
             tests
             linters
             search
             databases

The routing rule should be based on latency budget and data sensitivity.

A task such as “rename this variable” does not need a 744B model streamed from NVMe but a six-hour autonomous repository investigation may be a better fit.

I would also expose Colibrì through a queue rather than directly to product traffic.

Each job would have:

job_id workload_class cache_slot priority max_turns max_model_tokens wall_clock_deadline repository_snapshot tool_policy

The orchestrator would assign KV slots explicitly.

The bigger idea: sparse models create a storage hierarchy problem

Colibrì may or may not become the standard way to run giant MoE models but the important conclusion is that sparse models expose a new systems surface.

If only a small subset of a model is active for each token, then inference engineers can ask:

• Which weights must always be resident?
• Which weights can live on slower tiers?
• Can routing predict future I/O?
• Is expert popularity stable by workload?
• Can hot experts be promoted automatically?
• Can usage history survive restarts?
• Can RAM and VRAM act as explicit cache tiers?
• Can model state be persisted without re-prefill?
• When does speculative decoding increase storage traffic?
• Where does the bottleneck move after faster storage?

Colibrì already experiments with many of these questions.

Its PILOT=1 router-lookahead mode is particularly interesting.

Applying layer L+1’s router to layer L’s post-attention state recalls 71.6% of the true top-eight experts, compared with 41.3% for reusing the previous token’s experts.

A dedicated I/O thread can use that prediction to prefetch next-layer experts while the current layer computes.