Topic
Topic covering how language models are served in production: inference engines like vLLM and SGLang, GPU serving stacks, and long-context techniques.
OpenBMB's 2B-parameter model is Apache-2.0, speaks 30 languages, and serves through vLLM's OpenAI-compatible /v1/audio/speech. The real-time factor quoted for it was measured on a different backend than the one the project recommends for production.
Reality
NVIDIA's TensorRT backend now creates per-rank contexts and NCCL communicators inside one Triton instance, and the client sends a single gRPC request to a model name. The GPU count is fixed at compile time.
Reality
The Series F closed at a $30.9bn post-money valuation on one operating gigawatt and $140bn of contracted value spread across six. Turning the other five gigawatts into cash is what the new money bought.
Reality
The designated successor to GenAI-Perf splits load generation from record processing so the benchmark client stops hitting Python's GIL. Teams holding a GenAI-Perf baseline inherit a port and a re-run.
Reality
One endpoint fronts more than 300 models, but the company running the GPUs picks the inference engine and the quantization. A dev.to writeup says the quality gap that follows turns up in the response body, while the status code still reads success.
Reality
Ao Qu and collaborators open-sourced Reef on September 15th, an OpenAI-format inference server that stamps each response with a record ID, matches later feedback to that ID, and publishes a retrained artifact only if its evaluation stage accepts it.
Reality
The vLLM benchmark shows a finished MP4 arriving before its own playback would end, which is a different property from showing frames as they are made, and MiniMax's community licence still requires separate permission for US, EU, UK and Korean use.
Reality