Opening the post0.0s
12 min read
Agentic search stacks typically hand the model one lever: search(query).
When the agent can't find anything, it rewrites the query and tries again. It has no say over what happens after retrieval, meaning which candidates are kept, how they're ranked, and which passages come back as the observation.
A team from Tencent's Yuanbao and Peking University suggests this is where many search agents lose.
In their trace analysis on BrowseComp-Plus, 42.7% of verified supporting passages entered the extraction candidate set and were still never delivered to the agent.
The pipeline had the evidence and dropped it and this has real consequences for how you build agents, i.e., building the query as a unit that a small program can runs over a persistent pool of candidates.
Let's dive deeper.
If you've watched an agent "search" a codebase or a corpus, you've seen this: it issues query after query, slightly reworded, sometimes getting nowhere.
I've argued before that understanding a codebase is really a search problem dressed up as reasoning.
Programmatic Search Agents: Extending Agentic Search Beyond Query Reformulation puts numbers on the same problem for open-domain deep search.
The team traced Query-based Agent trajectories (DeepSeek-V4.1-Flash over a fixed retrieve → filter → rerank → extract pipeline) on BrowseComp-Plus.
That benchmark pairs hard research questions with a fixed corpus, so retriever and agent effects can be separated. They labeled a passage available once it entered the extraction candidates and delivered once it appeared in a returned search response.
Two findings stand out:
One caveat: partial omissions (some support already shown) showed no net success gain. Changing presentation matters most when the agent would otherwise see no supporting evidence at all.
The takeaway is that your bottleneck may not be retrieval quality or query quality. It may be the hard-coded funnel between the two, and the agent can't reach it.
The Programmatic Search Agent (PSA) borrows from CodeAct, which uses executable Python as an agent's action space, and from PyTerrier-style composable retrieval pipelines. The authors say it is directly inspired by Perplexity's Search as Code architecture.
At each step, the policy model generates a program cell in restricted Python, conditioned on the question, the history, and a compact description of the workspace. Three mechanisms do the work:
The five primitives, as defined in the paper:
| Primitive | Signature | Behavior |
|---|---|---|
retrieve | retrieve(q, k) → C | Up to k candidates for query q |
filter | filter(C, φ) → C′ | Keeps candidates satisfying predicate φ, original order |
rerank | rerank(C, k) → C′ | Up to k candidates, scored against the retrieval query stored in each candidate's provenance |
extract | extract(C, u, k) → E | Up to k evidence items per instruction u, source IDs preserved |
dedupe | dedupe(C₁‖…‖Cₘ, z) → C′ | Concatenates, keeps first candidate per key z, order preserved |
The key insight is about data dependencies versus judgment. Once the agent has decided "retrieve, then rerank, then extract," nothing about that chain needs another model call. A stepwise tool agent still pays a full LLM turn per dependent stage, because each stage consumes aliases returned by the previous one. PSA lets the runtime resolve those dependencies inside one cell and returns control only when new evidence actually needs interpreting.
This comparison is well controlled, all three agents share the same frozen search substrate: Qwen3-Embedding-0.6B retrieval with an exact inner-product index, Qwen3-Reranker-0.6B reranking, and deterministic passage-level extraction with Qwen3.5-0.8B. All three run under the same 20/10/10 per-call limits (retrieve ≤20, rerank keeps ≤10, extract picks ≤10 passages from the first 10 inputs) and a cap of 50 policy decisions per task.
| Query-based Agent | Tool-based Agent | PSA | |
|---|---|---|---|
| Decision unit | search(query) (parallel allowed) | One primary stage per turn | One program cell |
| Primitives | Hidden behind fixed funnel | Same 5 as PSA | 5 primitives |
| Persistent workspace | No | Yes (same as PSA) | Yes |
| Dependent stages | Fixed by pipeline | Separate model turns | Resolved inside cell |
The Tool-based Agent is the important control, it has PSA's primitives and PSA's workspace, so any gap between the two comes from the interface: composition and selective presentation in one action.
Benchmarks: the full BrowseComp-Plus set (830 questions) and InfoSeek-Eval (300 questions, the held-out split of InfoSeek). Five backbones were tested: DeepSeek-V4.1-Flash, Hunyuan-4-Preview, GLM-5.3-Flash, Qwen3.8-Flash, and GPT-5.6-Luna. None received task-specific training.
Macro-averaged across the five backbones:
| Metric | Query-based | Tool-based | PSA |
|---|---|---|---|
| InfoSeek-Eval task success | 77.53% | 79.73% | 81.53% |
| BrowseComp-Plus task success | 47.90% | 48.12% | 55.45% |
| BrowseComp-Plus supporting-doc Recall | 52.41% | 55.44% | 56.87% |
On final-step tokens, PSA comes in below both baselines for every backbone on both benchmarks. Relative to the Query-based Agent, it averages 28.3% fewer on InfoSeek-Eval and 33.9% fewer on BrowseComp-Plus. Against the Tool-based Agent, the reductions are 46.3% and 54.5%.
Three details from the per-backbone table are worth a closer look:
In a 128-task paired run with GLM-5.3-Flash, the team compared program-selected feedback against a full view of every extracted row. Selected feedback cut returned text by 37.2% (paired bootstrap 95% CI: 7.7–56.6%). The point estimates also showed 3.91 points higher success and 24.5% fewer final-step tokens (27.9K vs. 36.9K). Retaining a candidate and showing it to the model are separate decisions, and treating them separately pays off.
In a budget sweep with Qwen3.8-Flash on 128 BrowseComp-Plus tasks, PSA beat the Query-based Agent by 11.72–13.28 points at budgets of 30–50 decisions, using 38.3–40.9% fewer final-step tokens. The baseline's success stayed between 31.25% and 39.06% while its final-step tokens grew from 31.7K to 141.6K. More decisions only made its context longer.
There's an exception: at a 10-decision budget, PSA lost (35.16% vs. 39.06%). If your product enforces tight step limits, keep that in mind.
There is no code available but you have two real options:
Option A: Rebuild the paper's substrate. The components are public models: Qwen3-Embedding-0.6B with an exact inner-product index, Qwen3-Reranker-0.6B, and a small LLM doing deterministic passage-level extraction. BrowseComp-Plus provides the corpus and the scripts to plug your own retriever into deep-research agents.
Option B: Use the production analog. Perplexity's Search SDK is the closest thing you can install today. Its docs describe it as an agents-first Python SDK for composable retrieval primitives that agents orchestrate in code. The source is at perplexityai/perplexity-search-sdk, and the package is published on PyPI as pplx-srch-sdk:
pip install pplx-srch-sdkYou can use Agent Skill as the canonical reference for the SDK's methods, so get method names from there.
The code below is my own illustrative implementation of the pattern described in the paper. It is not the authors' code. It follows the paper's primitive signatures, the 20/10/10 limits, the σ(W) workspace projection, and the 50-decision budget. The backends (search_fn, score_fn, extract_fn) are placeholders for your own embedding index, reranker, and extractor.
from dataclasses import dataclass
@dataclass(frozen=True)
class Candidate:
doc_id: str
passage_id: str
text: str
query: str # retrieval-query provenance; rerank scores against this
class Workspace:
def __init__(self):
self.vars = {}
def describe(self) -> str:
"""sigma(W): names, types, sizes. Never contents."""
rows = []
for name, val in self.vars.items():
size = len(val) if hasattr(val, "__len__") else "-"
rows.append(f"{name}: {type(val).__name__}[{size}]")
return "\n".join(rows) or "(empty)"MAX_RETRIEVE, MAX_RERANK, MAX_EXTRACT = 20, 10, 10
def make_primitives(search_fn, score_fn, extract_fn):
def retrieve(q, k=MAX_RETRIEVE):
return search_fn(q, min(k, MAX_RETRIEVE)) # -> list[Candidate]
def filter_(C, pred):
return [c for c in C if pred(c)] # original order preserved
def rerank(C, k=MAX_RERANK):
ranked = sorted(C, key=lambda c: score_fn(c.query, c.text), reverse=True)
return ranked[:min(k, MAX_RERANK)]
def extract(C, instruction, k=MAX_EXTRACT):
rows = extract_fn(C[:MAX_EXTRACT], instruction) # dicts with passage_id, doc_id, note
return rows[:min(k, MAX_EXTRACT)]
def dedupe(*collections, key="passage_id"):
seen, out = set(), []
for C in collections:
for c in C:
if getattr(c, key) not in seen:
seen.add(getattr(c, key))
out.append(c)
return out
return {"retrieve": retrieve, "filter": filter_, "rerank": rerank,
"extract": extract, "dedupe": dedupe}rerank scores against each candidate's own retrieval query, as the paper specifies. That keeps the ranking objective explicit. Merging pools does not implicitly rerank them.
import ast
SAFE_BUILTINS = {"len": len, "range": range, "min": min, "max": max,
"sorted": sorted, "enumerate": enumerate, "list": list,
"dict": dict, "set": set, "str": str, "any": any, "all": all}
def run_cell(code: str, ws: Workspace, prims: dict) -> dict:
tree = ast.parse(code)
last = tree.body.pop() if tree.body and isinstance(tree.body[-1], ast.Expr) else None
env = {"__builtins__": SAFE_BUILTINS, **prims, **ws.vars}
try:
exec(compile(tree, "<cell>", "exec"), env)
out = eval(compile(ast.Expression(last.value), "<cell>", "eval"), env) if last else None
status = "ok"
except Exception as e:
out, status = None, f"error: {type(e).__name__}: {e}"
for k, v in env.items():
if k not in prims and k != "__builtins__":
ws.vars[k] = v # retained, not shown
return {"status": status, "observation": out}Using a single globals dict matters: list comprehensions inside the cell need to see cell variables. Also, trimming builtins isn't a security boundary. Run cells in a real sandbox with process-level timeouts and no network access except to your search backend.
def psa_loop(question, policy, ws, prims, max_decisions=50):
history = []
for _ in range(max_decisions):
action = policy(question=question, history=history,
workspace=ws.describe()) # your LLM call
if action["type"] == "finish":
return action["answer"]
result = run_cell(action["cell"], ws, prims)
history.append((action["cell"], result))
return NoneHere's a first cell (my example) that collects two query pools without dumping them into context:
a = retrieve("2019 film festival best debut feature director", 20)
b = retrieve("debut feature award director later television series", 20)
evidence_pool = dedupe(a, b, key="passage_id")
{"pool_size": len(evidence_pool)}And here is the abbreviated cell excerpt from the paper. It reprocesses a retained pool with no new retrieval (predicate and instruction abbreviate the original condition and relation):
focus = filter(evidence_pool, predicate)
ranked = rerank(focus)
grounded = extract(ranked, instruction)
result = {"ids": [r["passage_id"] for r in grounded],
"notes": [r["note"][:1600] for r in grounded[:3]]}
result # Return this view to the agent; retain workspace objects.Three dependent stages run in one decision. The model sees three notes and a list of IDs, while evidence_pool, focus, and grounded stay addressable for the next cell. That's the whole pattern.
"Final-step tokens" is not your bill. The paper measures the input messages plus output of the last decision call, averaged over tasks. That's a good proxy for context bloat, but it isn't total tokens, latency, or substrate compute. Model the whole loop with an agent cost and latency calculator before you promise anyone savings.
Watch the judge. GLM-5.3-Flash is the answer judge and also one of the five policy backbones. The paired evidence-feedback experiment used the same backbone for grading too. That's not disqualifying, but replicate with an independent judge before you rely on small deltas.
Selective presentation cuts both ways. A program that controls what the model sees can also hide contradicting evidence. Log the full workspace for each cell next to the observation, so you can audit what was retained but not shown. If you're mapping where this sits in your stack, it touches the tools, memory, and observability layers in the seven failure layers of agentic products.
Turn the delivery gap into a metric. The paper's available-vs-delivered instrumentation is something you can copy now, even with a fixed pipeline. Tag passages that reach extraction, tag what gets returned, and track recall between the two. Every "available but never delivered" trace is a regression test waiting to be written, which is exactly the approach in converting agent failures into evals.
Expect better cell generation later. The authors' proposed next step is trajectory-based self-distillation: reflect on completed runs, extract effective candidate-processing strategies, and distill them into reusable search programs. It's close in spirit to ReasoningBank's approach of mining trajectories for reusable strategies, applied to search programs instead of reasoning notes.
The pattern here is three ideas and about 100 lines of runtime: keep candidates as addressable state, let the model compose primitives in code, and make presentation a deliberate choice.
If your agent keeps rewording the same query, the evidence may already be in its candidate pool.