If you are building agents that touch private data, you have hit the same wall everyone hits: the frontier model lives in the cloud, the sensitive files live on the laptop, and moving one to the other is either a compliance problem or a latency problem.
Perplexity’s answer is Hybrid Compute, which landed in the Mac app on September 1, 2026.
Cloud models do the research, the planning and the long-horizon reasoning and a local model on the Mac handles the steps that read private files and drive local apps.
A local classifier also flags names, emails, account numbers and privileged files, and the user decides: keep it on the Mac, mask it, or send it anyway.
The catch is obvious once you build one of these.
The local hop becomes the bottleneck and if the on-device model needs 30 seconds to chew through a 4K-token file, the whole agent loop stalls.
So Perplexity wrote its own inference engine.
It is called Lily, it is specialized for exactly one model (Qwen3.6–35B-A3B) on exactly one hardware family (Apple Silicon), and on September 2 they open-sourced it under Apache-2.0.
They measured it on an M5 Max with a 40-core GPU and 128 GB of unified memory, batch size 1, averaged over ten prompt/context lengths from 256 to 128K tokens:
Metric Lily MLX-LM Ratio Prefill (tok/s) 4,156 3,388 1.23× Decode (tok/s) 170.0 126.4 1.35×
At a 4K prompt with a 4K context, Lily reached 5,749.9 prefill tok/s and 186.6 decode tok/s against MLX-LM’s 4,737.5 and 140.9.
Accuracy did not move.
In a teacher-forced comparison over 192 positions, Lily’s perplexity was 0.04% higher and it picked the same top token 96.35% of the time.
Editor’s note: If you want to dive deep into Local LLMs and Agentic Stack, you can join our Agent Foundry program for hands-on, in-depth trainings.
Let’s continue.
But what is Lily?
Lily is a single-process runtime.
A Rust layer loads the checkpoint, owns the session state and drives the generation loop.
A minimal OpenAI-compatible chat-completions server accepts requests and a hand-written Metal kernels execute the Qwen-specific operations.
Neither PyTorch nor MLX is in the execution path.
The Metal shaders compile from source at runtime, so there is no offline shader build step.
Compare that with the MLX-LM stack, where a Python model definition is expressed as composable MLX array ops that MLX schedules onto reusable kernels.
Those kernels have to serve dozens of architectures but Lily’s only have to serve one.
That narrowness is the entire performance argument and everything below follows from it.
Why Qwen3.6–35B-A3B is a weird workload
The model has 35B parameters, but only about 3B fire per token.
A router scores 256 expert sub-networks, picks 8, and adds one shared expert that sees every token.
It is also a hybrid.
10 full-attention layers are interleaved with 30 Gated DeltaNet layers.
The attention layers use grouped-query attention with 16 query heads and 2 KV heads, so 8 query heads share each KV head.
The DeltaNet layers compress history into a fixed-size recurrent state instead of a growing cache.
That gives you three irregular compute patterns in a single forward pass:
- Uneven expert groups (some experts receive many rows, some receive few)
- Attention over a KV cache that grows with context
- A fixed-size recurrence that can be evaluated as a sequential scan or reorganized into blocks
A general-purpose engine picks a reasonable kernel for each.
A single-model engine can tune tile sizes, memory layout and scheduling around the exact dimensions.
Prefill and decode are different problems
This is the mental model the whole engine is built on.
Prefill processes hundreds or thousands of prompt tokens at once.
Every weight block is reused across all those rows, so the linear layers run as matrix-matrix multiplies (GEMM). On M5, compatible GEMMs can hit the Neural Accelerators inside each GPU core through Metal 4 tensor ops.
Decode at batch 1 processes one new row per step. Each token needs another full pass over the weights with almost no reuse.
That is a matrix-vector multiply (GEMV), it is bound by memory bandwidth, and it belongs on the GPU’s vector ALUs.
Unified memory helps (weights stay resident, no separate GPU copy), but it does not make data movement free. Reading weights still burns bandwidth. So the prefill strategy is “maximize weight reuse and never wait on the CPU,” and the decode strategy is “minimize bytes moved per token.”
Prefill: keep routing on the GPU, dequantize in flight
The checkpoint uses groupwise affine 4-bit quantization.
Every group of 64 weights shares a bfloat16 scale and bias.
That shrinks roughly 70 GB of bfloat16 weights into a 19.4 GB checkpoint, which is what makes a 35B model resident on a laptop in the first place.
But Metal 4 tensor ops want bfloat16 operands, so the weights must be reconstructed before the multiply. The naive path expands Q4 into a bf16 array in unified memory, then the matmul reads it back.
Lily’s grouped GEMM dequantizes one small tile at a time inside threadgroup memory, accumulates in FP32, writes bf16, and never materializes the expanded array.
On a 512-token prompt, that alone lifted end-to-end prefill by 77.4%.
The bigger win was routing.
Grouped GEMM needs each expert’s rows stored together, which means histogram, prefix scan, scatter and block map after every top-8 selection.
The obvious implementation bounces to the CPU to inspect intermediate results and submit the next op. Lily keeps the whole sequence inside a single GPU command buffer per prompt chunk.
That adds two kernels and removes a CPU-GPU sync inside every MoE layer. Result: +89% prefill on a 512-token prompt.
Smaller wins: 32-row tiles with 4 simdgroups instead of 16-row tiles (+13.2% at 2K tokens, since a 2K prompt produces 16,384 token-expert assignments, roughly 64 rows per expert on average), and a register-resident Gated DeltaNet scan that keeps the recurrent state on-chip (+5.6% at 2K; the ablation path was moving 256 MiB of state per layer).
Long prompts are processed in bounded chunks, so temporary activations do not compete with weights and caches for memory, attention prefill stays quadratic in prompt length, chunking only caps peak working memory.
Decode: minimize bytes per token
Four groups of changes here.
- First, the token handoff never leaves the GPU. The runtime alternates between two command buffers and two GPU-resident token slots, GPU picks the argmax and writes the token ID directly into the next step’s input slot while the CPU prepares the following work.
- Second, concurrency. One recorded batch-1 decode step launched 795 GPU kernels whose dependencies formed 555 sequential stages. Metal’s default serial mode ran all 795 in order. Lily records real data dependencies into concurrent Metal passes, so independent kernels overlap and barriers appear only where a later op needs an earlier result.
- Third, fusion. Four kernel chains were fused: expert input projections with the gate activation, expert output projection with routing scores and the shared expert, Q/K preparation before attention, and the recurrent update with normalization. Each fused kernel keeps intermediates in registers, and each removed intermediate also removes a barrier.
- Fourth, the KV cache. Coalescing reads so adjacent threads request adjacent bytes pushed key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s (+2.1% decode at a 3,840-token context). GQA packing puts 4 query heads in one threadgroup so each KV row is loaded once and reused four times. 8 independent KV requests become 2 shared loads, worth +23.8% decode at 32K. Above 32K, the runtime switches to a fixed-block attention layout that spreads the cache scan evenly across the GPU: +7.7% at 32K, +27.4% at 64K, +40.2% at 128K.
What didn’t work
Speculative decoding made batch-1 decode 18% slower.
Verifying 2 to 5 draft rows at once is an awkward shape for this hardware, and the rows often route to different experts, which increases the expert weight bytes read.
Shrinking the drafter’s vocabulary sped up the drafter by about 5% but did not rescue the loop. (Perplexity notes that its batched Qwen deployment on Blackwell does use speculative decoding. Different regime, different answer.)
Fewer GPU launches, whole-step overlap, bigger prefill tiles, wider fusion, a faster router, fusing output projection with token selection: none of them improved the full inference loop.
And the ceiling is close. The MoE GEMM and GEMV reach 97.9% and 90.3% of the fastest sustained weight-read rate for their access patterns. Deleting the arithmetic from the sparse GEMV changed throughput by 0.2%. You are not compute-bound on decode.
You are bandwidth-bound, and you are nearly at the wall.
Setup
Requirements:
- Apple GPU family 10 or later (M5 and newer)
- macOS 26 or later (Metal 4 tensor operations)
- Rust 1.92, pinned by
rust-toolchain.toml - A local Qwen3.6–35B-A3B MLX affine 4-bit checkpoint with group size 64
Yes, that means M1 through M4 Macs are out for the open-source build.
Lily validates the exact 35B-A3B architecture and quantization layout at load time.
Dense Qwen, smaller Qwen, bf16, GGUF, AWQ, GPTQ, int8 and fp8 checkpoints are all rejected.
Pull the pinned checkpoint (19.4 GB):
hf download mlx-community/Qwen3.6-35B-A3B-4bit \
--revision 38740b847e4cb78f352aba30aa41c76e08e6eb46 \
--local-dir ~/models/Qwen3.6-35B-A3B-4bitBuild and run:
git clone https://github.com/perplexityai/pplx-garden.git
cd pplx-garden/lily
cargo build --release --locked
./target/release/lily \
--model ~/models/Qwen3.6-35B-A3B-4bit \
--bind 127.0.0.1:8000 \
--max-seq 4096--max-seq is prompt plus completion. The kernel limit is 262,144 tokens, clamped to the checkpoint's max_position_embeddings.
Three endpoints: POST /v1/chat/completions, GET /v1/models, GET /health.
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen3.6-35B-A3B",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 64,
"prompt_cache_key": "conversation-1"
}'The API is deliberately tiny
Read this before you wire Lily into an agent framework.
The server accepts text-only system/user/assistant messages, max_tokens, stream: false, and an optional prompt_cache_key. The last message must be user. Decoding is always greedy, and the chat template is always rendered with thinking disabled.
Everything else is rejected: sampling parameters, streaming, tools, response formats, multimodal content, speculative decoding.
So no tools=[...]. If you want function calling, you parse it out of the text yourself.
In practice that fits the role Lily plays in Hybrid Compute: a deterministic worker that reads private files and returns structured text, while the cloud model does the orchestration.
Here is the minimal Python client using the OpenAI SDK:
from openai import OpenAI
lily = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
def local_extract(doc: str, question: str, session: str) -> str:
resp = lily.chat.completions.create(
model="Qwen3.6-35B-A3B",
messages=[
{"role": "system", "content": "Answer only from the document. Reply in JSON."},
{"role": "user", "content": f"<document>\n{doc}\n</document>\n\n{question}"},
],
max_tokens=512,
stream=False,
extra_body={"prompt_cache_key": session},
)
cached = resp.usage.prompt_tokens_details.cached_tokens
print(f"reused {cached} prompt tokens")
return resp.choices[0].message.contentThe prefix cache is the agent feature
Multi-turn agent loops resend the same growing prefix on every call. Lily keeps a fixed two-entry LRU cache of decode states. A cached state is reused only when its token sequence is a strict prefix of the new prompt. prompt_cache_key picks which entry to prefer, but it never bypasses the token-equality check.
The response tells you how much you got back in usage.prompt_tokens_details.cached_tokens.
Two implications.
Keep your system prompt and document at the front and byte-stable across turns, or you will miss the cache every time. And with only two slots, do not interleave more than two conversations through one Lily process, or they will evict each other.
A hybrid router in ten lines:
def route(step, session):
# pii_gate is your local classifier. Perplexity open-sourced
# theirs as pplx-pii-masking on Hugging Face.
if pii_gate(step.context):
return local_extract(step.context, step.question, session)
return cloud.chat.completions.create(
model="your-frontier-model",
messages=step.messages,
).choices[0].message.contentReproducing the benchmark
The repo ships a fail-closed benchmark runner, and the measurement contract is worth reading even if you never run it: a fresh process per arm, an exact-shape warmup, one measured prefill, exactly 64 decode steps, deterministic synthetic prompt IDs, AC power required, and SHA-256 hashes of the binary, Cargo.lock, the runner and the MLX harness.
python3.12 -m venv .venv && . .venv/bin/activate
pip install -r benchmarks/requirements.txt
export LILY_BENCH_SOURCE_ID=$(git rev-parse HEAD)
cargo build --release --locked --bin lily-bench
python benchmarks/run_matrix.py \
~/models/Qwen3.6-35B-A3B-4bit \
target/release/lily-bench \
"$VIRTUAL_ENV/bin/python" \
./results \
--source-id "$LILY_BENCH_SOURCE_ID" \
--rounds 2 --decode-steps 64The second published report (September 2, against MLX 0.32.2 and mlx-lm 0.31.3) is more nuanced than the tweet.
Decode holds at 1.24× to 1.32× across every context from 256 to 128K. Prefill leads by 1.03× to 1.17× up to 16K, then flips: 0.99× at 32K, 0.90× at 64K, 0.80× at 128K. If your agent stuffs 100K tokens into the local model per call, MLX-LM currently prefills faster.
Perplexity published that table themselves, in the same repo as the engine. That is the rigor bar.
Concluding thoughts
Lily is not a general-purpose local runtime and does not pretend to be.
It is a demonstration of what you get when you stop abstracting over the model and the hardware: fused dequantization, GPU-resident routing, GQA packing and dependency-aware scheduling, each measured with an ablation, each with a number attached.
The broader bet is that high-performance local inference will increasingly come from engines tailored to a specific model and a specific chip, rather than engines that abstract the difference away.
With decode already at 90% of the memory wall, it is hard to argue the other side.
Nothing stops this train.