Local LLM Agents After the Harness Turn: Model + Loop Fit Beats Biggest Context

The findings for local builders:

• Bash-only reduced repeated patching of already edited files across all four models: from 3.3 to 0.4 re-patches for the 30B, 2.8 to 2.2 for the 120B, 4.6 to 1.5 for the 550B, 3.0 to 1.3 for Mistral.
• Predefined tools improved performance for models with weaker bash proficiency; bash-capable models operated effectively bash-only at substantially lower cost, especially on command-line-centric tasks.
• Planning shifted from an accuracy scaffold for the weaker model to a cost saver for stronger ones. Without planning, the 30B model's SWE-bench success rate fell to 13.60% from 25.20% with it.

The study did not measure frontier proprietary models, the newest open weights, or context windows in the 250k to 1M range. Commenters were right to say a Qwen or DeepSeek model from this quarter may behave differently. They were wrong to dismiss the result on that basis, because the result is "measure your model; the right harness depends on it", and that claim only gets stronger as models keep changing.

The two-by-two that replaces the model leaderboard

The shell-proficiency check is the input everything else depends on, so make it a real test rather than a vibe; the procedure is below. A model that re-patches the same file three times per task is telling you it needs tools with structure.

Context management is the local problem

The thread's other objection was that huge context windows make aggressive context management unnecessary. On a hosted frontier model, maybe. Locally, the window you can afford is bounded by memory and by the speed you are willing to tolerate at long context, and both bite well before a million tokens.

That makes the local agent the case where the harness study's context findings apply most, and it makes the eleven-system study's retrieval finding the cheapest win available. Production harnesses retrieve code with ripgrep, tree-sitter, glob and a Markdown context file at the root of the repository, and none of them uses embeddings. All of that is free, deterministic, and small in the window.

Then compact deliberately. The study's harnesses name their strategies, threshold compaction and lineage compaction among them, and a named strategy is a policy you chose before the window filled. For a local model with a 32k usable window, a policy that keeps the intent spec, the repository map and the last tool results, and summarises everything else at a threshold, will outperform a bigger model that is allowed to fill its window with transcript.

The opinionated local stack

Three notes on the choices.

Bash-first, not bash-only by decree. If the proficiency check says the model wanders, give it a small set of predefined tools with tight schemas. The study is explicit that this helps weaker models. Do not cargo-cult the bash-only result onto a model that cannot use it.

The proxy is not optional locally. A local agent with a shell and no policy is a root shell that reads untrusted files. Deny by default costs nothing and lets you say yes to the useful commands.

Measure before you switch models. The tempting move when a local agent struggles is to download a bigger model. Sometimes that is right. Often the study's result applies: the loop was wrong for the model, and the bigger model would have struggled the same way at twice the memory. The eval set tells you which.

Running the proficiency check

The matrix above depends on one input, whether your model can drive a shell.

Twenty tasks is enough to find out. Each one requires the model to locate a file with a shell tool, read it, edit it with a shell command rather than a structured edit tool, run a test, and interpret the result. Draw them from your own repository so the file layout and the test runner are the real ones. Vary them across the things that trip models up: paths with spaces, a file that must be edited in two places, a test that fails for a reason unrelated to the edit, a command whose output is long.

Score three things per task. Whether the task succeeded on the first attempt, judged by the test. How many times the model re-patched a file it had already edited, which is the study's tell for a model flailing at the shell. And the token cost. Run each task three times so a lucky first attempt does not decide the matrix.

A re-patch count near the study's bash-only figures for capable models, well under two per task, with a pass^3 you can live with, puts the model in the top row of the matrix. A re-patch count near three or above, or a pass^3 that collapses, puts it in the bottom row, where predefined tools with tight schemas and planning are the scaffold the study found helps.

Repeat the check whenever the model changes, and a model change includes a quantisation change. A different quantisation of the same weights is a different model for this purpose, since harness effects are conditional on the model in front of them.

A minimal loop in sixty lines

Production harnesses run on hand-rolled loops, and the local case is where writing your own pays off most, because the loop is where you fit the model. This is the shape, stripped to four things: a bash-first action space, a budget, verify-on-stop, and every command through a policy decision.

Every design choice in it comes from one of the two studies. The action space is bash because the model passed the proficiency check; a model that did not would get a small set of structured tools here instead. The compaction is rules first, summarisation second, at a threshold, with pinned items that survive, which is the configuration the controlled study found most efficient. The loop cannot end on the model's say-so; it ends when the tests pass or the budget runs out, and it records which. And the shell never runs a command the policy has not seen.

It is the thing to measure a product against, and for a single engineer with a single repository it is often enough.

Compaction as a policy

The controlled study's context findings contradict a habit.

The habit is to treat compaction as an emergency measure that fires when the window is nearly full, and to make it as lossless as possible by summarising carefully. The study found the benefit of context management concentrates in preventing overflow failures, that staging rule-based elision before LLM-based summarisation gave the best efficiency, and that making elided content recoverable produced no accuracy gain. So drop the obviously droppable things by rule, early and cheaply, summarise what is left, and do not build machinery to get dropped content back, because the model does not benefit from it.

For a local model with a small window, that means a compaction policy you can state in three lines. Tool output is truncated to a tail at the moment it is produced. Old turns are elided at a fixed threshold, keeping the pinned files and the recent turns. What remains is summarised once. The window stays comfortably under its limit, the model never sees an overflow, and you have spent nothing on recoverability.

Where the enterprise study lands

The same question, whether bash is enough, is now being asked outside coding. A separate paper this year, "Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents", asks it for agents doing enterprise work rather than software engineering. We have not run its experiments and will not quote its numbers here, but its existence means the question has become a research programme rather than a Hacker News argument. If your local agent is a digital worker rather than a coding agent, that is the paper to read next, with the same rule: measure on your model, on your tasks.

Which local model, then

The question the post refused to lead with, answered in the order the evidence supports.

First, run the proficiency check on the models your hardware can serve at the quantisation you will use. Our earlier posts on running coding agents in 48GB of VRAM, on the Qwen and GLM families, and on outperforming the hosted harnesses for local workflows are the practical starting points for what fits where. Second, pick the loop from the matrix for each model that passes. Third, build the eval set from your own tasks and run the candidates through it, with cost per accepted outcome as the deciding number. Fourth, and only fourth, decide whether the bigger model is worth its memory.

Teams that go in the other order download the biggest model that fits, wrap it in a framework's loop, and conclude local agents do not work. Both studies suggest they concluded the wrong thing about the wrong component.

Where to start

The loop is the part you own, and it decides whether the model your hardware can serve gets the work done, so start there rather than at the download page. Run the twenty-task proficiency check on the model already on your machine, three attempts per task, at the quantisation you serve, and pick the row of the matrix it lands in. Put the proxy in front of the shell before the first real task, with deny as the default. Write the compaction policy in three lines, pin the intent spec and the repository map, and point retrieval at ripgrep instead of an embedding index. Then, and only then, argue about which model to download. Once the loop and proxy are set, ReLiveGym's work on wake-up timing for long-lived agents shows the trigger policy deserves the same deliberate attention.