6.2GB for Qwen3.8-27B. Local LLM Tax Just Dropped Again.

We want to briefly share a very critical update for Local LLM inference.

Unsloth released Dynamic 3.0 GGUFs, which is the next iteration of their Dynamic quantization.

Qwen3.8–27B now reaches roughly 6.2GB at the lowest quant that retains around 72% top-1% accuracy yet being 89% smaller, pushing a 27B model into hardware envelopes that used to be reserved for much smaller models.

However it’s important to note that GGUF file size is not total runtime memory. KV cache, context, runtime buffers, and GPU offload still need headroom.

You can download the new UD-* quant and run it normally.

What changed from previous version

Dynamic 3.0 preserves more of the original model’s behavior at similar file sizes than its previous quantization approach.

The practical change is simple: new weights with same runtimes.

Here are practical starting points:

Extremely memory constrained: UD-IQ1_M
Around 8GB for weights: UD-IQ2_S
16GB-class machines: UD-IQ4_XS
Quality-first local workstation: UD-Q4_K_M or UD-Q6_K

The lower you quantize, the more aggressively you trade model fidelity for memory.

Editor’s note: If you want to master Local LLMs and Agentic Stack, join our Agent Foundry program with hands-on and in-depth trainings.

Quick start with llama.cpp

Unsloth exposes the new quants directly through Hugging Face.

Install llama.cpp on macOS/Linux:

Code

bash

curl -LsSf https://llama.app/install.sh | sh

Then run the model directly:

llama cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Or expose it as a local OpenAI-compatible server:

llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

llama.cpp serves the OpenAI-compatible API on port 8080 by default.

A minimal Python client looks like this:

Code

python

from openai import OpenAI

client = OpenAI(
  base_url="http://localhost:8080/v1",
  api_key="no-key",
)

response = client.chat.completions.create(
  model="unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M",
  messages=[
      {"role": "system", "content": "You are a software engineering agent."},
      {"role": "user", "content": "Review this patch and identify failure modes."},
  ],
)

print(response.choices[0].message.content)

This is the important architectural point for agentic systems: keep your agent interface stable and swap the local model underneath it.

Your tool loop, memory layer, repo indexing, and orchestration do not need to care whether inference comes from a cloud API or a local GGUF.

Using it with a coding agent

Unsloth provides a ready-made example for the Pi coding agent.

Start llama.cpp:

llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Install Pi:

Code

bash

npm install -g @mariozechner/pi-coding-agent

Then configure ~/.pi/agent/models.json:

Code

json

{
"providers": {
  "llama-cpp": {
    "baseUrl": "http://localhost:8080/v1",
    "api": "openai-completions",
    "apiKey": "none",
    "models": [
      {
        "id": "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M"
      }
    ]
  }
}
}

Run it inside your repository:

That is already enough to test a local coding-agent workflow against the new quant.

LM Studio setup

You do not enable Dynamic 3.0 inside LM Studio.

Download the new GGUF instead:

Code

bash


lms get unsloth/Qwen3.8-27B-GGUF --gguf  --always-show-download-options

Choose one of the new UD-* quantizations, then load it normally from LM Studio.

You can also start LM Studio’s local API server:

Code

bash


lms server start

LM Studio exposes OpenAI-compatible endpoints, so the same agent code can usually be reused by changing the base URL to:

http://localhost:1234/v1

Dynamic 3.0 is not MTP

This distinction is easy to miss.

Dynamic 3.0 = quantization quality / lower memory.

MTP = faster generation.

They solve different problems.

If you only download the new Dynamic 3.0 GGUF, you are getting the new quantized weights but you are not enabling some hidden inference acceleration mode.

Also note that Unsloth currently distributes a separate MTP module for the smallest quants where it is not embedded.

Native MLX versions of these new Dynamic 3.0 files are not available yet.

The important part here is the deployment boundary moved.

A 27B model can now be tested on machines where it previously would have been dismissed before the first benchmark ran.

But do not benchmark an old GGUF and assume you tested Dynamic 3.0, re-download the new UD-* file, run the exact same agent workflow, and compare quality, latency, and memory on your own workload.

You can find more details from official announcement.