Local LLM Agents After the Harness Turn: Model + Loop Fit Beats Biggest Context

For two years the local LLM conversation has been about which model to run. That was the right question when the harness was an afterthought. It is the wrong question now, because the harness studies of 2026 showed the loop around the model changes the answer.

The controlled study that set off the Hacker News debate tested three harness components, context management, planning and the action space, across four models and two long-horizon coding benchmarks, in 176 matched settings. Its finding is one sentence: the effect of each component is conditional on the model. Bash-capable models did better and cheaper with a bash-only interface; weaker models needed predefined tools. Planning was an accuracy scaffold for the small model and a cost saver for the big ones.

For a local agent, where the model is the one your hardware can run rather than the one you would pick, that is the whole design brief. Fit the loop to the model.

What the study measured, and what it did not

Be precise about the evidence, because the Hacker News thread was not.

The paper is An Empirical Study of Harness Design for Coding Agents, submitted September 17, 2026. Four models: Nemotron-3 at 30B, 120B and 550B, and Mistral-Medium-3.5-128B. Two benchmarks: SWE-bench Verified and Terminal-Bench 2.1. Three components varied: context management, planning, and action space (bash-only versus predefined tools). 176 matched settings, so the comparisons are paired rather than averaged across different task mixes.

Findings that matter for local builders:

• Bash-only reduced repeated patching of already edited files across all four models: from 3.3 to 0.4 re-patches for the 30B, 2.8 to 2.2 for the 120B, 4.6 to 1.5 for the 550B, 3.0 to 1.3 for Mistral.
• Predefined tools improved performance for models with weaker bash proficiency; bash-capable models operated effectively bash-only at substantially lower cost, especially on command-line-centric tasks.
• Planning shifted from an accuracy scaffold for the weaker model to a cost saver for stronger ones. Without planning, the 30B model's SWE-bench success rate fell to 13.60% from 25.20% with it.

What it did not measure: frontier proprietary models, the newest open weights, or context windows in the 250k to 1M range. Commenters were right to say a Qwen or DeepSeek model from this quarter may behave differently. They were wrong to dismiss the result on that basis, because the result is not "use bash". The result is "measure your model; the right harness depends on it". That claim gets stronger, not weaker, when models keep changing.

The two-by-two that replaces the model leaderboard

The shell-proficiency check is the input everything else depends on, so make it a real test rather than a vibe. Twenty tasks that require reading a file, editing it with a shell tool, running a test and interpreting the output. Score single-attempt success and the re-patch count. A model that re-patches the same file three times per task is telling you it needs tools with structure.

Context management is the local problem

The thread's other objection was that huge context windows make aggressive context management unnecessary. On a hosted frontier model, maybe. Locally, the window you can afford is bounded by memory and by the speed you are willing to tolerate at long context, and both bite well before a million tokens.

That makes the local agent the case where the harness study's context findings apply most, and it makes the eleven-system study's retrieval finding the cheapest win available. Production harnesses do not retrieve code with embeddings. They use ripgrep, tree-sitter, glob and a Markdown context file at the root of the repository. All of that is free, deterministic, and small in the window.

# The retrieval layer production harnesses actually use, in three lines.
rg -n "def handle_webhook" --type py            # find the symbol, not the similar text
rg -l "handle_webhook" | head                    # who calls it
sed -n '1,60p' AGENTS.md                         # the repository map the harness reads first

Then compact deliberately. The study's harnesses name their strategies, threshold compaction and lineage compaction among them, and the point of naming them is that compaction is a policy you choose, not something that happens to you when the window fills. For a local model with a 32k usable window, a policy that keeps the intent spec, the repository map and the last tool results, and summarises everything else at a threshold, will outperform a bigger model that is allowed to fill its window with transcript.

The opinionated local stack

Three notes on the choices.

Bash-first, not bash-only by decree. If the proficiency check says the model wanders, give it a small set of predefined tools with tight schemas. The study is explicit that this helps weaker models. Do not cargo-cult the bash-only result onto a model that cannot use it.

The proxy is not optional locally. A local agent with a shell and no policy is a root shell that reads untrusted files. Deny by default costs nothing and lets you say yes to the useful commands.

Measure before you switch models. The tempting move when a local agent struggles is to download a bigger model. Sometimes that is right. Often the study's result applies: the loop was wrong for the model, and the bigger model would have struggled the same way at twice the memory. The eval set tells you which.

Running the proficiency check

The matrix above depends on one input, whether your model can drive a shell, so make that a measurement rather than an impression.

Twenty tasks is enough to find out. Each one requires the model to locate a file with a shell tool, read it, edit it with a shell command rather than a structured edit tool, run a test, and interpret the result. Draw them from your own repository so the file layout and the test runner are the real ones. Vary them across the things that trip models up: paths with spaces, a file that must be edited in two places, a test that fails for a reason unrelated to the edit, a command whose output is long.

Score three things per task. Whether the task succeeded on the first attempt, judged by the test. How many times the model re-patched a file it had already edited, which is the study's tell for a model flailing at the shell. And the token cost. Run each task three times so a lucky first attempt does not decide the matrix.

# Score a proficiency run. One row per (task, attempt) in runs.jsonl:
# {"task": "t07", "attempt": 2, "passed": true, "edits": ["src/a.py", "src/a.py", "src/b.py"], "tokens": 18420}
import json, collections

rows = [json.loads(l) for l in open("runs.jsonl")]
by_task = collections.defaultdict(list)
for r in rows:
  by_task[r["task"]].append(r)

def repatches(edits):               # edits to a file after its first edit
  seen, n = set(), 0
  for f in edits:
      n += f in seen
      seen.add(f)
  return n

first_try = sum(any(a["passed"] for a in v if a["attempt"] == 1) for v in by_task.values()) / len(by_task)
every_try = sum(all(a["passed"] for a in v) for v in by_task.values()) / len(by_task)
avg_repatch = sum(repatches(r["edits"]) for r in rows) / len(rows)
avg_tokens = sum(r["tokens"] for r in rows) / len(rows)
print(f"pass@1 {first_try:.2f}  pass^3 {every_try:.2f}  re-patches/task {avg_repatch:.2f}  tokens/task {avg_tokens:.0f}")

The reading: a re-patch count near the study's bash-only figures for capable models, well under two per task, with a pass^3 you can live with, puts the model in the top row of the matrix. A re-patch count near three or above, or a pass^3 that collapses, puts it in the bottom row, where predefined tools with tight schemas and planning are the scaffold the study found helps.

Repeat the check whenever the model changes, and a model change includes a quantisation change. A different quantisation of the same weights is a different model for this purpose; the study's whole point is that harness effects are conditional on the model in front of them.

A minimal loop in sixty lines

Production harnesses run on hand-rolled loops, and the local case is where writing your own pays off most, because the loop is where you fit the model. This is the shape, stripped to what matters: a bash-first action space, a budget, verify-on-stop, and every command through a policy decision.

import subprocess, json

BUDGET = {"steps": 40, "tokens": 120_000}
PINNED = ["intent.md", "AGENTS.md"]          # survive compaction

def policy(cmd: str) -> str:                  # allow | deny | ask, from the proxy or a local table
  return proxy.decide("shell.exec", {"command": cmd})

def run(cmd: str) -> str:
  out = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=120)
  return (out.stdout + out.stderr)[-4000:]  # tail; the model does not need 40k lines of logs

def compact(msgs):                            # rules first, summarise second
  keep = [m for m in msgs if m["role"] == "system" or m.get("pinned")]
  recent = msgs[-12:]
  older = [m for m in msgs if m not in keep and m not in recent]
  summary = llm.summarise(older) if older else None
  return keep + ([{"role": "user", "content": f"[summary of earlier work]\n{summary}"}] if summary else []) + recent

msgs = [{"role": "system", "content": SYSTEM}] + [{"role": "user", "content": open(p).read(), "pinned": True} for p in PINNED]
msgs.append({"role": "user", "content": TASK})
used = {"steps": 0, "tokens": 0}

while used["steps"] < BUDGET["steps"] and used["tokens"] < BUDGET["tokens"]:
  if llm.count(msgs) > 0.85 * WINDOW:
      msgs = compact(msgs)
  reply = llm.chat(msgs)                    # the model answers with either a shell command or DONE
  used["steps"] += 1; used["tokens"] += reply.usage
  msgs.append({"role": "assistant", "content": reply.text})
  if reply.text.strip() == "DONE":
      verdict = run("pytest -x -q")         # verify-on-stop: the loop may not end on the model's word
      if "failed" not in verdict:
          break
      msgs.append({"role": "user", "content": f"Tests still fail:\n{verdict}"})
      continue
  cmd = reply.text.strip()
  decision = policy(cmd)
  if decision == "deny":
      msgs.append({"role": "user", "content": f"Denied by policy: {cmd}"}); continue
  if decision == "ask" and not approve(cmd):
      msgs.append({"role": "user", "content": f"Not approved: {cmd}"}); continue
  msgs.append({"role": "user", "content": run(cmd)})

record = {"model": llm.model, "quant": llm.quant, "loop": "v3", "used": used, "result": "verified" if used["steps"] < BUDGET["steps"] else "budget"}
json.dump(record, open("run.json", "w"))

Every design choice in it comes from one of the two studies. The action space is bash because the model passed the proficiency check; a model that did not would get a small set of structured tools here instead. The compaction is rules first, summarisation second, at a threshold, with pinned items that survive, which is the configuration the controlled study found most efficient. The loop cannot end on the model's say-so; it ends when the tests pass or the budget runs out, and it records which. And the shell never runs a command the policy has not seen.

It is not a product. It is the thing to measure a product against, and for a single engineer with a single repository it is often enough.

Compaction as a policy

The controlled study's context findings deserve their own paragraph for the local case, because they contradict a habit.

The habit is to treat compaction as an emergency measure that fires when the window is nearly full, and to make it as lossless as possible by summarising carefully. The study found the benefit of context management concentrates in preventing overflow failures, that staging rule-based elision before LLM-based summarisation gave the best efficiency, and that making elided content recoverable produced no accuracy gain. In other words: drop the obviously droppable things by rule, early and cheaply; summarise what is left; and do not build machinery to get dropped content back, because the model does not benefit from it.

For a local model with a small window, that means a compaction policy you can state in three lines. Tool output is truncated to a tail at the moment it is produced. Old turns are elided at a fixed threshold, keeping the pinned files and the recent turns. What remains is summarised once. The window stays comfortably under its limit, the model never sees an overflow, and you have spent nothing on recoverability.

Where the enterprise study lands

The two coding-agent studies are not the only ones asking whether bash is enough. A separate paper this year, "Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents", asks the same question for agents doing enterprise work rather than software engineering. We have not run its experiments and will not quote its numbers here; the reason to mention it is that the question has become a research programme rather than a Hacker News argument. If your local agent is a digital worker rather than a coding agent, that is the paper to read next, with the same rule: measure on your model, on your tasks.

Which local model, then

The question the post refused to lead with, answered in the order the evidence supports.

First, run the proficiency check on the models your hardware can serve at the quantisation you will actually use. Our earlier posts on running coding agents in 48GB of VRAM, on the Qwen and GLM families, and on outperforming the hosted harnesses for local workflows are the practical starting points for what fits where. Second, pick the loop from the matrix for each model that passes. Third, build the eval set from your own tasks and run the candidates through it, with cost per accepted outcome as the deciding number. Fourth, and only fourth, decide whether the bigger model is worth its memory.

Teams that go in the other order download the biggest model that fits, wrap it in a framework's loop, and conclude local agents do not work. Both studies suggest they concluded the wrong thing about the wrong component.

What changed, in one paragraph

Before the harness turn, a local agent was a model plus whatever loop you copied from a framework. After it, the loop is the part you own and the part that decides whether your model can do the work. Fit the action space to the model's shell proficiency. Turn planning on only if the model needs it. Retrieve deterministically. Compact by policy. Put a proxy in front. Then, and only then, argue about which model to download.