Verdict
Stay on chunked prefill until your workload crosses concrete thresholds that NVIDIA itself publishes: its Dynamo disaggregated-serving guide states that for balanced workloads with input:output (ISL:OSL) ratios between roughly 2:1 and 10:1, aggregated serving is often the better choice, and that disaggregation shines for very long inputs (ISL above ~8,000 tokens) with short outputs, high concurrency, and multi-node deployments with RDMA networking. Chunked prefill is free - it is enabled by default in vLLM V1 and tuned with a single knob - while disaggregation buys its headline gains (DistServe's up to 7.4x goodput, Dynamo's claimed 30x for DeepSeek-R1 on GB200 NVL72) at the price of KV-cache transfer over the network, a second pool of GPUs to capacity-plan, and a router that must get prefill/decode ratios right. For most single-node and small multi-GPU deployments serving agent or chat traffic, the honest expected win from disaggregation is the 1.4-1.8x range TensorRT-LLM's disaggregated-serving tech blog reports for representative DeepSeek-R1 traffic (ISL 4400 / OSL 1200, MTP off) - real, but not worth two pools until SLOs force it.
The two architectures attack the same problem - prefill and decode contend for the same GPU - from opposite directions. Chunked prefill (introduced by the Sarathi-Serve paper, arXiv 2403.02310) breaks long prompts into scheduler-step-sized chunks and batches them with decode tokens: vLLM V1 enables it by default and controls the budget with max_num_batched_tokens, where the docs' guidance is ~2048 for better ITL and >8192 for optimal throughput (docs.vllm.ai optimization guide; the pre-V1 default was 512). A TNG Tech / Hugging Face benchmark write-up measured chunked prefill turning full decode stalls into a slowdown - and raising total token throughput by about 50% in a standard vLLM deployment - at the cost of a small TTFT increase for the chunked request, a tradeoff vLLM's own upstreaming RFC (vllm issue #3130) states plainly: ITL improves, TTFT and e2e latency get slightly worse. Disaggregation eliminates the interference instead of softening it: DistServe (OSDI '24, arXiv 2401.09670) showed that separating prefill and decode onto different GPUs yields up to 7.4x more requests served or 12.6x tighter SLOs at the same hardware, precisely because TTFT and TPOT stop competing. Production systems followed: SGLang's DeepSeek P/D deployment on 96 H100 GPUs reported 52.3k input tok/s and 22.3k output tok/s per node for 2k-token inputs - prefill benchmarked on 4 nodes (EP32) and decode on 9 nodes (EP72), per the lmsys.org write-up - and NVIDIA claims up to 30x more requests for DeepSeek-R1 on GB200 NVL72 and >2x for Llama 70B on Hopper with Dynamo (developer.nvidia.com Dynamo announcement). But every one of those wins is a multi-node, high-concurrency, MoE-or-long-context regime; NVIDIA's own disaggregated-serving guide (docs.nvidia.com/dynamo) is the rare vendor doc that tells you when NOT to use its product, and its thresholds are the most useful decision data in this comparison.
Decision Table
| Criterion | Edge | Explanation |
|---|---|---|
| TTFT/TPOT isolation under mixed long-prompt + interactive load | Prefill/decode disaggregation (separate GPU pools: NVIDIA Dynamo, SGLang P/D, vLLM disaggregated serving, llm-d) Best edge | This is the problem disaggregation actually solves. With co-located serving, a 30k-token prefill steals compute from every in-flight decode; chunked prefill softens this to a slowdown but cannot remove it - vLLM's chunked-prefill RFC concedes ITL still degrades because decode tokens share batches with prefill chunks. DistServe's OSDI '24 results quantify what full isolation buys: up to 7.4x more requests under the same TTFT/TPOT SLOs, or 12.6x tighter SLOs at the same rate, with >90% of requests inside latency constraints (arXiv 2401.09670). If you have hard per-token latency SLAs (voice agents, interactive coding assistants) alongside long-document prefill traffic, separate pools are the only architecture that makes the two workloads stop interfering. |
| Small deployments (1-2 nodes, no RDMA) | Chunked prefill on a co-located pool (the vLLM/SGLang single-deployment default) Best edge | Disaggregation's overhead is structural: KV cache produced by the prefill pool must move to the decode pool, and NVIDIA's Dynamo docs state that without RDMA-capable networking (InfiniBand/RoCE, or NVLink within a rack) the transfer itself becomes the bottleneck and performance degrades sharply. You also split your GPUs into two fixed-purpose pools, which strands capacity whenever the prefill:decode ratio drifts from your provisioning. On a single node or a couple of PCIe boxes, chunked prefill delivers most of the interference relief with zero transfer cost - the TNG Tech/HF write-up's ~50% throughput gain came from flipping a config flag, not re-architecting. Vultr's Dynamo cookbook reaches the same conclusion: disaggregation is situational on single-node and shines multi-node. |
| Prefill-heavy, long-context workloads (RAG over big documents, agent context replay) | Prefill/decode disaggregation (separate GPU pools: NVIDIA Dynamo, SGLang P/D, vLLM disaggregated serving, llm-d) Best edge | NVIDIA's disaggregated-serving guide gives the concrete threshold: disaggregation pays off for very long input sequences (ISL above roughly 8,000 tokens) with short outputs - the regime where prefill dominates GPU time and starves decode. Dynamo 0.4 reports up to 4x faster interactivity for gpt-oss-120b on B200 at very long ISL, and the GB200 NVL72 DeepSeek-R1 result (up to 30x requests served, per NVIDIA's Dynamo launch blog) is exactly this shape: massive MoE prefill separated from latency-bound decode. Agent platforms that re-send 50k-token contexts every turn are in this regime more often than they think - check your ISL:OSL ratio before dismissing it. |
| Balanced chat/agent traffic (ISL:OSL between 2:1 and 10:1) | Chunked prefill on a co-located pool (the vLLM/SGLang single-deployment default) Best edge | Straight from the vendor selling the disaggregation stack: NVIDIA's Dynamo disaggregated-serving guide says that for balanced workloads with ISL:OSL between about 2:1 and 10:1, aggregated serving is often the better choice. Most chat and tool-calling agent traffic lands here - a 3k-token context producing a 500-token reply is 6:1. In this band, chunked prefill's single-knob tuning (max_num_batched_tokens ~2048 for latency, >8192 for throughput, per vLLM's optimization docs) captures the available win, and the KV-transfer plus dual-pool overhead of disaggregation buys little. TensorRT-LLM's disaggregation tech blog reports 1.4-1.8x speedups for DeepSeek-R1 at ISL 4400 / OSL 1200 without MTP - useful at scale, not transformative on a handful of GPUs. |
| Rack-scale and multi-node throughput economics | Prefill/decode disaggregation (separate GPU pools: NVIDIA Dynamo, SGLang P/D, vLLM disaggregated serving, llm-d) Best edge | Every headline disaggregation result is multi-node: SGLang's DeepSeek-style deployment on 96 H100s hit 52.3k input / 22.3k output tok/s per node (prefill benchmarked on 4 nodes, decode on 9); NVIDIA's Dynamo 0.4 reports 2.5x higher throughput per GPU for DeepSeek-R1 671B on GB200 NVL72 with SLO-based autoscaling; and the llm-d project (Red Hat, Google, CoreWeave backing) builds P/D disaggregation into Kubernetes-native serving because at fleet scale, independently scaling prefill and decode capacity is a capacity-planning win, not just a latency win. If you operate tens of GPUs with RDMA interconnect, disaggregation is where the per-GPU economics are heading - all three major open stacks (vLLM, SGLang, Dynamo) now treat it as first-class. |
| Operational complexity and failure surface | Chunked prefill on a co-located pool (the vLLM/SGLang single-deployment default) Best edge | Chunked prefill is a scheduler behavior inside one engine process - it cannot half-fail. Disaggregation adds: a router that classifies and dispatches requests, KV-cache transfer (NIXL/NCCL paths in Dynamo, bootstrap-server handshakes in SGLang P/D), two autoscaling groups whose ratio must track workload shape, and new partial-failure modes (prefill pool healthy, decode pool saturated, transfers timing out). vLLM's own docs long marked its disaggregated prefill support as experimental while the chunked-prefill path became the always-on default in V1 - a fair proxy for relative maturity. Budget SRE time for the second architecture; none for the first. |
| Mid-scale scaling path (4-16 GPUs, growing traffic) | Tie | At this scale a third option usually beats both sophisticated answers: plain data-parallel replicas of aggregated engines behind a prefix-cache-aware router. Each replica runs default chunked prefill; the router (llm-d's inference gateway, SGLang's router, or a plain load balancer with session affinity) sends repeat-context traffic to the replica that already holds the prefix KV. You get horizontal scaling, zero KV-transfer machinery, cache reuse for agent loops that replay context every turn - and you keep the option to re-shard into P/D pools later. The published disaggregation wins (DistServe's 7.4x, Dynamo's 30x) were measured against single-replica baselines or at rack scale, not against well-routed replica fleets; for 4-16 GPUs the replica fleet is the defensible default. |
Choose Prefill/decode disaggregation (separate GPU pools: NVIDIA Dynamo, SGLang P/D, vLLM disaggregated serving, llm-d) if...
- Multi-node fleets (tens of GPUs+) with RDMA interconnect serving mixed long-context and interactive traffic under strict TTFT and TPOT SLOs.
- Prefill-dominated workloads: ISL above ~8k tokens with short outputs - large-document RAG, agent platforms replaying big contexts every turn.
- Large MoE models (DeepSeek-class) where prefill and decode want different parallelism strategies - the regime behind SGLang's 96-H100 P/D deployment and NVIDIA's GB200 results.
- Teams on Kubernetes planning for independent prefill/decode autoscaling - Dynamo and llm-d both make P/D pools a first-class scheduling primitive.
Choose Chunked prefill on a co-located pool (the vLLM/SGLang single-deployment default) if...
- Single-node and small multi-GPU deployments - chunked prefill is on by default in vLLM V1 and captures most of the interference relief with one tunable.
- Balanced chat/agent traffic in NVIDIA's own 2:1-10:1 ISL:OSL band, where the vendor's guide says aggregated serving is often better.
- Teams without RDMA networking, where KV-cache transfer would bottleneck a disaggregated setup.
- Anyone who wants throughput now without new failure modes: the measured ~50% token-throughput gain from enabling chunked prefill (TNG Tech/Hugging Face benchmark) costs a config flag, not an architecture.
Decision rules
- 01Measure your ISL:OSL ratio first. NVIDIA's own Dynamo guide says aggregated serving is often better between 2:1 and 10:1, and disaggregation shines above ~8,000 input tokens with short outputs - if you don't know your ratio, you can't make this decision.
- 02No RDMA (InfiniBand/RoCE/NVLink between pools) = no disaggregation. NVIDIA's docs state KV-cache transfer over plain Ethernet can become the bottleneck and erase the gains.
- 03On 1-2 nodes, tune chunked prefill before considering anything else: vLLM V1 has it on by default; set max_num_batched_tokens ~2048 if ITL/TPOT is your pain, >8192 if throughput is (vLLM optimization docs).
- 04If you have hard TPOT SLOs AND long-prompt traffic in the same deployment and chunked-prefill tuning can't hold both, that is the disaggregation trigger - it is the only architecture where the two stop competing (DistServe: up to 7.4x goodput under joint SLOs).
- 05At 4-16 GPUs, try prefix-cache-aware data-parallel replicas before P/D pools - agent workloads that replay context get cache hits that neither chunked prefill nor disaggregation gives you by itself.
- 06Discount vendor multipliers to workload reality: Dynamo's 30x is DeepSeek-R1 on GB200 NVL72 rack hardware; TensorRT-LLM's own tech blog reports 1.4-1.8x for representative DeepSeek-R1 traffic (ISL 4400 / OSL 1200, MTP off). Pilot with your traffic shape before committing GPUs to fixed pools.
Migration Notes
- Instrument before migrating: per-request ISL, OSL, TTFT, TPOT percentiles, and queue time. The disaggregation decision is entirely a function of these distributions, and you will need the same dashboards to validate the migration.
- Migrate incrementally: run a P/D pool alongside your aggregated fleet and route only the long-ISL slice (e.g. ISL > 8k) to it first - that traffic sees the biggest win and the smallest regression risk.
- Get prefill:decode pool sizing from data, not defaults: SGLang's reference evaluation benchmarked the two phases at different scales (prefill on 4 nodes at EP32, decode on 9 nodes at EP72) because the phases want different parallelism - your ratio follows your ISL:OSL mix and drifts with product changes, so make it autoscalable (Dynamo 0.4's SLO-based autoscaler exists precisely for this).
- Keep chunked prefill enabled on the decode pool anyway - disaggregated systems still chunk residual prefill work, and the two techniques compose rather than compete.