EdgeAgent: Local Multi-Agent Stack Is Leaving Half Your Mac Idle
Agent Native•October 6, 2026
Suppose you're building an agent that works with private files.
You moved inference onto the laptop for privacy, split the work into an orchestrator plus a few workers, and turned on speculative decoding.
It still feels slow.
The model is probably fine but the problem is that the inference engines we use locally were built for one user making one request at a time, and a multi-agent workflow doesn't look like that.
EdgeAgent takes this problem apart carefully.
On Apple M4, there is a 1.29x speedup from UMA-aware execution alone, compared with batched speculative decoding. The full system reaches 1.77x under extreme tool-use latency.
Let's have a look at why the problem exists and what's the solution.
Three ways multi-agent workloads break edge inference
1. The UMA memory wall
On Apple silicon, the CPU and GPU share one memory pool. MLX's unified memory docs put it plainly: "The CPU and GPU have direct access to the same memory pool."
That makes zero-copy sharing possible, it also means both processors compete for the same bus.
Micro-benchmarks on the M4 show what happens when CPU and GPU run together:
Matrix (K×N)
CPU GB/s
GPU GB/s
Co-run GB/s
Co-run vs GPU
4096×4096
66.1
54.3
73.9
1.36x
4096×14336
68.9
80.3
91.3
1.14x
4096×128256
72.0
96.7
95.4
0.99x
On the vocabulary-sized matrix, the GPU alone already uses about 80% of the 120 GB/s peak. Adding the CPU makes things slightly worse, so splitting decode naively across CPU and GPU gives you almost nothing.
2. Some agents are much harder to draft for than others
Speculative decoding helps because it loads the target model's weights once for several candidate tokens.
How much it helps depends on how predictable the output is.
EdgeAgent's M4 experiment with DeepSeek-R1-Distill-Llama-8B, structured output peaks at 1.69x with draft length 16 and then gets worse as rejections pile up.
Reasoning tasks hit diminishing returns at much shorter draft lengths.
In a multi-agent system, your orchestrator writes open-ended reasoning, which is hard to draft for.
Your workers produce code and tool-call JSON, which are easy.
Current speculative decoding systems use one draft length for the whole batch, that wastes bandwidth on the orchestrator's rejected drafts and limits the workers.
If a stalled agent still holds slots in the batch, those slots do nothing. Static batching assumes every sequence is always ready to run, and agent workloads break that assumption.
This is already visible in shipping tools. LM Studio's MLX engine fails with SpeculativeDecodingNotSupportedError when you combine a draft model with batched MLX models. "Batched" and "speculative" are still treated as features you can't use together, and multi-agent workloads need both.
Setup: what you can run today
Step 1: Check that your CPU has SME
EdgeAgent's CPU path depends on Arm's Scalable Matrix Extension (SME). The Hello SME paper reports that the M4 was the first chip to support SME.
If you're on M1 through M3, EdgeAgent's CPU kernels don't apply to your machine.
Step 2: See how MLX handles CPU/GPU dependencies
The MLX docs show the programming model. Arrays live in unified memory, and you pick the device per operation instead of moving data:
Code
text
a = mx.random.normal((100,))
b = mx.random.normal((100,))
mx.add(a, b, stream=mx.cpu)
mx.add(a, b, stream=mx.gpu)
When one stream depends on another, MLX inserts the synchronization for you:
Code
text
c = mx.add(a, b, stream=mx.cpu)
d = mx.add(a, c, stream=mx.gpu)
This automatic tracking works at the level of whole tensors, and EdgeAgent runs into its limits. When the CPU and GPU write separate column ranges of the same buffer, the graph can't tell the writes don't overlap. It treats them as a conflict and adds conservative synchronization, or it forces a concat that copies both halves. I cover the fix below.
Step 3: Get a baseline with speculative decoding in mlx-lm
The mlx-lm server supports speculative decoding through --draft-model and --num-draft-tokens. Measure with and without these flags before you assume they help. One mlx-lm issue reports a 35% throughput drop on a MoE target (Qwen3.5-397B-A17B, with Qwen3.5-9B as the draft) because 17B active parameters is too close to the draft model's size. It's the same lesson as EdgeAgent: speculation only pays off when the workload suits it.
Layer 1: Zero-copy tensor parallelism across CPU and GPU
EdgeAgent fixes partitioning when the model loads:
Every linear projection (MLP gate/up/down, attention Q/K/V/O) is split along output columns. The CPU gets n = α·N columns and the GPU gets the rest. α is computed per layer from K/N so both devices finish at about the same time.
The attention core is split by heads. The GPU runs fused SDPA over its heads and the CPU runs a grouped matmul over the rest. The two results are then concatenated along the head axis.
Only one copy of the weights exists. GPU shards stay in linear SIMT layout. CPU shards are repacked once into 16×64 FP16 SME tiles.
Because each device writes its own columns, you never need a reduction across devices. Here's a NumPy sketch of the idea (mine, not the paper's):
Code
text
import numpy as np
K, N = 4096, 14336 # an MLP up-projection shape from the paper's Table 1
alpha = 0.4 # ASSUMPTION: paper derives alpha per layer from K/N
n = int(alpha * N)
x = np.random.randn(1, K).astype(np.float32) # one decode token
W = np.random.randn(K, N).astype(np.float32)
W_cpu, W_gpu = W[:, :n], W[:, n:] # split on output columns
out = np.empty((1, N), dtype=np.float32) # one preallocated buffer
out[:, :n] = x @ W_cpu # "CPU" writes its columns
out[:, n:] = x @ W_gpu # "GPU" writes the rest
print(np.allclose(out, x @ W, rtol=1e-4, atol=1e-3)) # True: no reduction needed
The hard part is the graph.
EdgeAgent replaces concat with a Logical Dependency Barrier. It's a node that makes downstream operators wait for both producers but allocates no memory, and it simply aliases the shared buffer.
Heterogeneous Fast Path then runs that subgraph outside MLX's generic scheduler, without cross-device fences.
Metal's memory model is weak, so the authors don't depend on fine-grained coherence. No address is ever written by both devices, so there's no data race. The barrier supplies the one happens-before ordering the design needs.
On the CPU side, the SME kernels use 16×64 micro-tiles, multi-vector loads (svld1_x4), an active set kept within the 8 MB effective L2, and lock-free work-stealing. The work-stealing keeps slower E-cores from holding back the P-cores.
Layer 2: HAL-based draft budgets
Profiling shows SME matmul performance rising in steps at multiples of 16 and dropping off after 16. The global draft budget is therefore fixed at M = 16 slots, shared across all agents.
Each agent gets a Historical Accepted Length (HAL): an EMA of the absolute number of tokens accepted per verification step. The authors chose absolute counts over acceptance rate because a shallow draft can have a perfect acceptance rate while producing very few tokens.
Update: A_i(t) = γ·k_i(t) + (1−γ)·A_i(t−1), with γ = 0.3 and A_i(0) = 4.0 chosen by grid search
Allocation: every agent gets L_min slots, and the remaining slots are split by a temperature-scaled softmax over HAL, rounded down
Here it is as runnable Python. The paper doesn't give values for L_MIN or TAU, so I've flagged both as assumptions:
Code
text
# hal.py: reimplementation of EdgeAgent Sec. 3.2.1 equations (not official code)
import math
from dataclasses import dataclass
L_TOTAL = 16 # paper: hardware-aligned global draft budget
GAMMA = 0.3 # paper: EMA decay factor
A0 = 4.0 # paper: initial HAL prior
L_MIN = 1 # ASSUMPTION: paper does not state its value
TAU = 1.0 # ASSUMPTION: paper does not state its value
@dataclass
class Agent:
name: str
hal: float = A0
suspended: bool = False
def update_hal(agent: Agent, accepted: int) -> None:
# A_i(t) = gamma * k_i(t) + (1 - gamma) * A_i(t-1)
agent.hal = GAMMA * accepted + (1 - GAMMA) * agent.hal
def allocate(agents: list[Agent]) -> dict[str, int]:
active = [a for a in agents if not a.suspended]
n = len(active)
if n == 0:
return {}
residual = L_TOTAL - n * L_MIN
assert residual >= 0, "more active agents than L_TOTAL / L_MIN"
z = sum(math.exp(a.hal / TAU) for a in active)
return {
a.name: L_MIN + math.floor(residual * math.exp(a.hal / TAU) / z)
for a in active
}
Layer 3: Suspend-and-yield on tool calls
At the end of each verification step, the scheduler checks every agent's state. An agent waiting on a tool moves to a suspended queue, and its slots go back to the pool for the allocator to hand out again. Its KV cache is frozen in place in shared memory. When the tool returns, the agent resumes without re-running prefill.
Quick start: simulate the scheduler
This adds tool stalls to the allocator above. The acceptance trace is made up for illustration:
Code
text
# sim.py
from hal import Agent, allocate, update_hal
agents = [Agent("orchestrator"), Agent("coder"), Agent("tool_worker")]
# Toy acceptance counts per verification step (NOT from the paper)
trace = {
"orchestrator": [1, 2, 1, 1, 2, 1],
"coder": [6, 7, 8, 7, 8, 8],
"tool_worker": [5, 6, 6, 0, 0, 6],
}
tool_stall_steps = {3, 4} # tool_worker waits on an external tool
for t in range(6):
agents[2].suspended = t in tool_stall_steps # suspend-and-yield
budget = allocate(agents)
print(f"step {t}: {budget} used={sum(budget.values())}/16")
for a in agents:
if not a.suspended:
update_hal(a, min(trace[a.name][t], budget[a.name]))
The simulation shows three things the paper doesn't discuss:
Rounding down wastes slots. The formula as written leaves 1–2 of the 16 slots unused per step in this run. A real implementation probably hands out the remainder, but the paper's formula doesn't say so.
τ has a big effect. With τ = 1, the softmax heavily favors one agent: the coder gets 14 slots during the stall. Tune τ against your own traces.
A suspended agent's HAL stays frozen. When tool_worker comes back, it competes using its pre-stall HAL. That works here, but if a tool result changes what the agent generates next (for example, from JSON to free text), its HAL will be out of date for a few steps.
How to read the numbers
1.29x is an ablation: the execution layer alone, compared with batched speculative decoding.
1.77x is the full system under extreme tool latency. That's the condition where suspend-and-yield gains the most, so expect less if your agents rarely wait on tools.
The workload is synthetic. The authors mix LongBench, MBPP and ToolBench prompts in a 1:2:2 ratio (reasoning : code : tool) because no deterministic multi-agent benchmark exists. They explain the ratio: with 1:1:1, a single reasoning agent tends to be left working through the queue alone at the end.
My excerpt doesn't include head-to-head results against llama.cpp or stock MLX. Don't claim EdgeAgent beats them.
What to change in your agent stack today
You can't pip install EdgeAgent, but its conclusions apply now:
Set draft length per agent, not per process. Your orchestrator and your JSON-producing workers need different speculation settings. If your runtime has only one global setting, run them as separate model instances.
Make tool calls give up their compute. If your orchestration layer holds an inference slot open while waiting on a tool, you're paying for idle time. Questions about decomposition, like whether you need multiple agents at all, come first. When multi-agent setups help and when they hurt covers that tradeoff.
Don't assume CPU+GPU helps during decode. On UMA chips, adding a second processor to bandwidth-bound decode can make it slower (0.99x in the table above). Co-execution pays off in prefill. In decode, it only works when combined with speculation that raises arithmetic intensity.
Track absolute accepted tokens, not acceptance rate. Even without EdgeAgent, logging HAL per agent tells you which roles are worth speculating on.
Open questions
Will code be released? Without a repo, nobody can reproduce the results.
Hardware coverage. Only M4 and M4 Pro were tested. The paper mentions Snapdragon X as a UMA platform but doesn't evaluate it.
Model size and precision. 8B in FP16 on 32 GB is a realistic setup. Most local deployments use quantized weights, and those change the bandwidth math.
What happens past 16 agents? With L_min ≥ 1, more than 16 concurrent agents can't fit in the budget. The excerpt doesn't describe admission control.
Frameworks. The paper cites LangGraph and OpenClaw as the kind of multi-agent frameworks it targets. Hooking a runtime like this into a gateway-style setup such as OpenClaw's routing and agent architecture would require the "cross-layer APIs" the authors mention but don't spell out.
