Most teams optimizing Mixture-of-Experts training still think in components but Cursor took a different route.
Their Mixture-of-Kittens (MoK) now powers training for its Composer models across tens of thousands of GPUs.
It provides up to 2.37x higher MXFP8 forward throughput than the fastest public baseline in Cursor’s single-layer benchmarks on GB300 NVL72s.
Cursor reports that replacing its previous DeepEP-based MoE stack with MoK moved end-to-end training throughput from 760.9 to 1,070.2 tokens/second/GPU across a 512-GPU production benchmark.
That is 1.41x, measured after contact with everything else in a real training run!
The interesting part is that MoK changes the unit of optimization.
Instead of treating expert routing, token movement, FFN execution, synchronization, quantization, and backward replay as separate pieces, MoK treats the entire expert-parallel MoE layer as one scheduling problem.
As models, clusters, and training loops get more complex, the next big speedups will come less from optimizing individual operators and more from deleting the boundaries between them.
Let’s dive a littler deeper.
What Cursor Actually Open-Sourced
Mixture-of-Kittens is a fully deterministic MoE training megakernel designed specifically for NVIDIA Blackwell NVL72 systems.
It supports BF16 and MXFP8 training, forward and backward passes, expert-parallel routing, fused communication and expert computation, configurable compute/communication overlap, PyTorch symmetric memory, and bitwise deterministic execution.
Cursor built MoK for DeepSeek-V3-style MoE layers: one shared expert plus many routed experts selected by a top-k router.
That architecture shows up across several modern open-weight model families, which is why the repo is more interesting than a one-off optimization for a single internal model.
Editor’s note: If you want to dive deep into Local LLMs and Agentic Stack, you can join our Agent Foundry program for hands-on, in-depth trainings.
Why MoE Training Stops Being a Kernel Problem
In a dense transformer layer, most of your optimization attention can stay on matrix multiplication but MoE changes that.
Suppose your model has hundreds of routed experts spread across an expert-parallel group.
A router picks the top-k experts for every token, and the selected experts may live on different GPUs.
A straightforward implementation becomes:
router
-> dispatch tokens to expert GPUs
-> run expert FFNs
-> send expert outputs back
-> weighted combineDispatch and combine are all-to-all communication.
If communication and compute run sequentially, you pay for both on the critical path, so every modern MoE stack tries to overlap them.
The problem is that building “overlap communication and compute” means answering all of this:
- Who pushes or pulls each remote token?
- How do tokens arrive in expert-friendly order?
- How large should each communication chunk be?
- How many SMs should communication occupy?
- When is a destination buffer safe to reuse?
- How do you avoid CPU synchronization when routing is dynamic?
- How do you preserve determinism when many GPU tasks execute concurrently?
- How do you leave room for inter-rack traffic such as FSDP all-gathers?
MoK’s architecture is essentially Cursor’s answer to all of those questions at once.
Design Choice 1: Pull to Dispatch, Push to Combine
A common assumption in GPU networking is that push-based communication should win, because it moves less protocol metadata.
Cursor found that this is not always the right choice for MoE routing on an NVL72.
For forward dispatch, MoK uses a pull-based scheme: the destination GPU pulls the token data it needs from the source GPUs.
Because a push schedule is surprisingly expensive to construct correctly.
For every outgoing token, the source needs to know the destination rank and the exact destination index.
The schedule also has to interleave destinations to keep NVLink lanes busy while laying tokens out in expert order so grouped GEMMs can start immediately.
With pull-based dispatch, the schedule collapses to something much simpler: the destination already knows which source rank and source token index it needs.
Cursor says the device-side scheduling kernel ends up taking less than 3% of total MoE runtime.
More importantly, the same schedule can be reused for all four communication directions in a training pass:
- Forward dispatch: pull
- Forward combine: push
- Backward reverse-combine: pull
- Backward reverse-dispatch: push
That symmetry eliminates a surprising amount of signaling complexity.
Cursor’s microbenchmarks put push-based forward dispatch signaling at roughly 103 microseconds versus 18 microseconds for pull-based signaling in their multi-node setup, about a 5.8x difference.
The fastest primitive in isolation does not necessarily produce the fastest system. Push moves fewer bytes; pull still wins because scheduling and synchronization dominate the critical path.
If you have ever benchmarked a component, shipped it, and watched end-to-end latency refuse to move, this is the same failure mode at a different altitude.
Design Choice 2: Make Overlap Granularity a Tuning Knob
There are two obvious ways to overlap communication with expert compute.
You can go extremely fine-grained: move a tiny tile, compute on it, move the next tile, or you can go coarse: move thousands of tokens, then run one large GEMM.
MoK does neither by default.
Cursor found the best point sits in the middle, and that it depends on model shape, token count, network bandwidth, and how many SMs are assigned to communication.
MoK calls each chunk a minibatch, and the repository exposes minibatch_size as a first-class tuning parameter rather than hiding the overlap policy behind a fixed implementation.
That matters because the tradeoff is physical.
If minibatches are too small, tensor cores repeatedly hit synchronization points before enough work has accumulated to saturate them.
If minibatches are too large, compute waits too long for the first tokens to arrive, and communication waits too long for processed outputs.
Cursor’s heuristic is worth writing on a sticky note: choose a minibatch large enough for the expert grouped-GEMM to sustain at least two waves of work across the SMs.
In the example published in the MoK post, their Kimi 2.5-shaped forward microbenchmark improved from 5.981 ms at minibatch 512 to roughly 3.425 ms at minibatch 2560, after which larger batches stopped helping materially.
This is exactly why you should not treat the default config as magic.
MoK is fast partly because it exposes the parameters that matter instead of guessing on your behalf.
Design Choice 3: Partition the SMs Yourself
MoK splits the GPU into two logical groups: compute SMs run the expert FFNs, and communication SMs run dispatch and combine.
The two groups coordinate through local counters inside the megakernel.
Forward roughly behaves like this:
comm SMs: pull mb0 ---- pull mb1 ---- pull mb2 ---- combine... compute SMs: FFN mb0 ----- FFN mb1 ----- FFN mb2...
The shared expert can run while compute SMs are still waiting for the first routed minibatch, so the bubble at the start of the layer does useful work.
Cursor says it tried multiple streams with green contexts, but found software partitioning more reliable for allocating SMs exactly as intended.
That is another recurring pattern in high-performance AI infrastructure: eventually, orchestration itself becomes part of the kernel.
Design Choice 4: The Ring Buffer That Deletes the CPU From the Loop
MoE routing is dynamic.
Before the router runs, you do not know how many tokens each rank will receive.
A simple implementation has two options, and both are bad.
- You can allocate a fixed maximum-sized destination buffer and drop overflow tokens, which hurts training quality.
- Or you can compute token counts on the GPU, copy them to the CPU, allocate exact buffers, and continue, which introduces a synchronization point.
Cursor specifically calls the second option expensive on GB300 NVL72 systems, where its traces showed GPU streams catching up to CPU-side work and going idle.
MoK avoids both with a fixed-size ring token buffer that Cursor calls a macrobatch.
Instead of allocating for the theoretical worst case, MoK continuously reuses a bounded buffer at minibatch granularity. As soon as a region is consumed and combined, that region becomes available for the next dispatch.
At macrobatch boundaries, MoK interleaves the combine of the previous macrobatch with the dispatch of the next one into the same ring region.
This is a scheduling technique that removes a CPU decision from the critical path.
Many “GPU performance” problems are actually host-orchestration problems wearing a CUDA hat.
“Megakernel” Is the Load-Bearing Word
The term megakernel has become increasingly relevant in AI systems.
The basic idea is to move scheduling inside a persistent GPU kernel instead of bouncing between many separate kernel launches.
Kernel boundaries introduce bubbles, and a persistent software scheduler can expose overlap that normal launch boundaries hide.
MoK applies that idea to MoE training.
Instead of launching separate kernels for dispatch, grouped GEMMs, activations, combine, and orchestration, the megakernel schedules tasks at the SM level.
That buys two things:
- It removes repeated launch boundaries around minibatch and macrobatch orchestration
- It gives Cursor deterministic control over which SMs are doing communication versus compute.
MoK is a different execution model for the entire MoE layer.
Determinism Is a Performance Feature
MoK is bitwise deterministic.
Cursor says it fixed the order of floating-point operations so identical inputs produce identical outputs regardless of hardware scheduling or instruction issue order.
Cursor built it for internal ablations and on-policy RL post-training, where agentic models rely on complex loops in which tiny system-level differences make debugging enormously harder.
If you are changing reward logic, rollout generation, routing behavior, or data mixtures, deterministic execution gives you a clean baseline for the only question that matters:
Did the model change because of the experiment, or because the execution order changed underneath it?
For training infrastructure, reproducibility is throughput.
Every non-reproducible regression costs you a debugging cycle you could have spent on the model.
MXFP8 Is Built Into the Dataflow
MoK supports both BF16 and MXFP8.
Cursor trains with MXFP8 while keeping the shared expert in BF16 for stability.
The interesting implementation choice is where quantization happens.
Routed expert weights are prequantized, while activation quantization is fused into the existing data path: dispatch, grouped GEMMs, and SwiGLU.
That reduces extra memory traffic and avoids turning quantization into another standalone stage.
Same pattern as everywhere else in this codebase: eliminate boundaries.
The Benchmarks
Cursor reports the following peak speedups over the fastest public baseline for each tested mode, on GB300 NVL72s:
- MXFP8 forward 2.37x
- MXFP8 backward 1.78x
- BF16 forward 1.92x
- BF16 backward 1.58x
These are standalone MoE-layer benchmarks covering the full layer path: routing schedule, dispatch, expert FFNs, combine, and final weighted sum. The public-baseline set included NCCL + PyTorch, DeepEP + PyTorch, DeepEP + TransformerEngine, and HybridEP + Megatron.
The published setup used expert parallel degree 64 and 2,048 tokens per GPU before routing, across four model shapes corresponding to Kimi K2.7 Code, GLM-5.2, Qwen3.5–397B-A17B, and DeepSeek-V4-Pro style MoE dimensions.
The production benchmark is a separate thing entirely.
Cursor’s previous production stack used DeepEP for expert-parallel communication with custom MXFP8 compute kernels.
On 512 GPUs across multiple GB300 NVL72 racks:
- Previous DeepEP-based stack: 760.9 tokens/sec/GPU
- MoK: 1,070.2 tokens/sec/GPU
- Improvement is 1.41x
The 2.37x is a best-case standalone forward result and the 1.41x is what survived contact with the rest of a real training system, which makes it the number infrastructure engineers should actually plan around.
MoK For Agentic Workflows
If you are building an agent product, you may never touch MoK directly.
Most teams do not own an NVL72 rack but the architecture points toward five trends that will reach your stack anyway.
1. Infrastructure optimization is getting more important
Long-horizon coding agents, tool-using models, and RL-trained agent systems consume enormous training and post-training workloads.
A 41% end-to-end throughput gain changes how quickly a fixed cluster can iterate.
When your model-development loop depends on repeated rollouts, ablations, and post-training runs, wall-clock speed is product velocity.
2. The boundary between communication libraries and compute kernels is dissolving
Traditional stacks have clean layers:
framework -> collective library -> kernel library -> hardwareMoK deliberately crosses those boundaries, scheduling communication and compute together, because the global optimum cannot always be expressed as the sum of locally optimal components.
Expect more of this, and expect the same pressure to show up one level up in your own agent runtime, where planning, tool execution, and inference are still artificially separated stages.
3. Determinism is becoming a first-class systems primitive
For on-policy RL and agent training, reproducibility is not just nice for papers.
It is how you debug reward changes, data changes, routing changes, and infrastructure changes without mixing them together.
MoK’s bitwise determinism signals that performance infrastructure is starting to absorb requirements that used to live at the experiment layer.
4. Hardware-specific software is back
MoK is not portable in the generic “runs on any CUDA GPU” sense.
It is aggressively designed around Blackwell features, NVL72 topology, TMA, symmetric memory, and CLC.
The largest gains often appear exactly when software stops pretending every accelerator looks the same.
5. Agents are changing kernel engineering itself
AI coding agents are increasingly capable of producing performant low-level kernels when given the right design direction.
The MoK team says it used agents to work through the megakernel’s complexity directly, rather than building another abstraction framework around the problem.
Agentic coding models are now helping engineers write the kernels that make training future agentic coding models faster.
That loop is unlikely to slow down.
What MoK Does Not Mean
It is easy to overread a release like this, so here are the guardrails.
It does not mean DeepEP is obsolete. DeepEP remains a high-performance expert-parallel communication library with a much broader role as a reusable communication layer. MoK makes a different tradeoff: it fuses communication and compute into one hardware-specialized execution model.
It does not mean every MoE workload gets 2.37x. The published number is “up to” 2.37x for MXFP8 forward across a particular benchmark suite and hardware target.
It does not mean you can reproduce the 1.41x production result from the public repo alone. Cursor open-sourced the MoE implementation and layer benchmark code, its full internal training stack is not the artifact being released.
And it definitely does not mean MoK is ready for your Hopper cluster or arbitrary cloud topology. The repository explicitly requires Blackwell SM100/SM103 and targets GB200/GB300 NVL72 systems. That hardware specificity should be part of your adoption decision from day one.
Concluding Thoughts
The biggest idea in Mixture-of-Kittens is not MXFP8. It is not pull-based dispatch. It is not the ring buffer. It is not even the megakernel by itself.
The real idea is that MoE performance is an orchestration problem. Cursor got its gains by looking at the full path:
routing
-> scheduling
-> remote token movement
-> tensor-core utilization
-> synchronization
-> buffer lifetime
-> quantization
-> backward replay
-> scale-out coexistenceThen it optimized that path as a single system, that is why this release is worth reading even if you never install it.
As agentic AI systems scale, the easy operator-level wins are getting harvested fast. The remaining performance is increasingly trapped between abstractions: between CPU and GPU, between compute and network, between one kernel and the next, between a communication library and a model runtime.
MoK is a concrete example of what happens when you delete those boundaries.
And the same question now applies one layer up, where most of us actually work: where is the performance trapped between the abstractions in your stack?
- Between your retriever and your model
- Between your planner and your tool executor
- Between your queue and your GPU
Most agent systems are still assembled from locally optimal parts.
If you have found one of those seams and closed it, I would like to hear how it went, leave a comment with the boundary you deleted and what it bought you.