Topic
Technique for splitting model training across many GPUs or nodes via data, tensor, or expert parallelism, aided by checkpointing and memory optimizations.
Ai2's Olmo-core 3 reached about 52,000 tokens per second per GPU on a 47B mixture-of-experts model, roughly 2.7 times the old stack, on eight B300 GPUs. The report labels every headline rate as measured under random routing, so a team with a trained router has to rerun the comparison.
Reality
Ai2 released Olmo-core 3, an open mixture-of-experts training stack it has benchmarked at over one trillion total parameters. Its speedups are measured against Ai2's own earlier FSDP code, so teams on Megatron-Core must run their own comparison.
Perspective Coverage
3 publishersReality
Halo adds expert and tensor parallelism to Hugging Face models and still saves SafeTensors that from_pretrained can load. Its best number, 9,009 tokens per second per GPU against TRL's 3,885, came from synthetic fixed-length sequences.
Reality
AWS's NVRx walkthrough puts synchronous checkpointing ahead of GPU faults as the source of idle time on its FSDP jobs, and pairs a background save with a restart that re-enters training without cycling the container.
Reality
A JAX and Transformer Engine version of the same GB200 run reports 1,068 TFLOPS per GPU, a 10.4x gain. The 84% share bounds what fixing communication alone could have bought, so the expert matmuls got faster too.
Reality
A custom SGLang framework feeds teacher outputs into the loss while training runs. The reported eightfold speedup comes from a stack of separate optimizations, and the named multipliers reach about 5.5x on their own.
Reality