Topic
Practice of measuring AI inference systems' throughput and latency across hardware, quantization, and load levels, emphasizing reproducible reporting.
Amazon's concurrency sweeps push rising traffic at a SageMaker inference endpoint and report where throughput stops improving. Three vLLM settings in the sample deployment decide whether that curve transfers to your traffic.
Reality
AWS counted 13 SageMaker inference launches so far in 2026. The one that changes production behaviour most is a prioritized list of up to five instance types, each allowed its own model optimization settings.
Reality
The ONNX export alone ran 1.58x faster than eager PyTorch with bit-identical accuracy. The int8 pass after it cost 8% in latency and bought a 64 MB download, small enough to serve as a static page with no backend.
Reality
Two arms on the same laptop differ by one flag. The small card wins because llama.cpp leaves Gemma 4's 1.93 GB per-layer embedding table in mmap and pulls a few rows per token, so only about 1.08 GB of body is resident.
Reality
The 1.7x to 3.6x latency range is set by the baseline systems, not the chip, and the report's own publication date is unsettled. Read it as direction, not evidence.
Perspective Coverage
9 publishersReality
NVIDIA reports 2.5x more concurrent users for Nemotron 3 Ultra from its packaged NIM serving stack. The gain comes from five interacting layers measured on one agentic traffic shape, and NVIDIA ships the harness to retest it.
Reality
nano-vLLM's decode cost fits in one expression, W/B plus KV(Tavg), and a batch-1 context sweep on Qwen3-0.6B shows why those two terms move in opposite directions as you add requests or lengthen prompts.
Reality