Topic
Running large language models on consumer GPUs and Apple Silicon with llama.cpp, Ollama and LM Studio.
Nemotron 3.5 Super VL, NVIDIA's 120B model, had GGUF, MLX, FP8 and EXL3 conversions on Hugging Face a day after its October 15 release. The roughly 128 GB memory figure comes from a DEV Community write-up whose author did not run the model.
Reality
The Neuron measured NVIDIA's Nemotron 3.5 Bolt 74B-A9B at 65.5 tokens per second on a laptop in a hands-on test. The build was heavily compressed, and the review's tool-assisted wins went to the hosted Super VL.
Reality
Apple says four Mac Studios run Meta's Muse Spark Open at over 36 tokens per second, Computerworld reports. The speed is Apple's own measurement, and the cost case is the columnist's inference from the price of four machines.
Reality
Apple says a cluster of four Mac Studios linked with RDMA over Thunderbolt generates over 36 tokens per second from an 816-billion-parameter model. The figure is Apple's own, from its MLX LM benchmark tool, and it covers generation speed only.
Reality
Janas, a C engine by Maurizio Cammalleri, reports 37 tok/s on a 48.4 GB model with 32 GB of RAM, against 2.8 on llama.cpp. Speculative drafting supplies 13 of those 37, so the like-for-like figure is 24 tok/s.
Reality
Ollama's OpenAI-compatible endpoint reported 3,875 prompt tokens per second where the real rate was 125, a dev.to author found. Any benchmark that repeats a prompt and reads that field overstates prefill speed by the share served from cache.
Reality
GitHub added local Ollama model discovery to Copilot CLI 1.0.94-0 and took its agent sandbox to general availability on October 7, dev.to reported. Keeping work on a developer's machine still takes manual steps, since Ollama must be installed first and offline mode is a separate setting.
Reality
TornadoVM's University of Manchester team says its jitLLM engine runs LLM inference on NVIDIA GPUs from Java at about 90% of llama.cpp's speed. The figure comes from the project's own RTX 5090 recordings, so a JVM team still needs a run on its own GPUs before it retires a Python sidecar.
Reality
Alibaba's new 27B open-weight model defaults to its deepest reasoning setting. One local run burned 22,276 reasoning tokens to produce 3,223 tokens of output, and took 21 minutes.
Perspective Coverage
6 publishersReality
VIDRAFT released a 111GB, 4-bit build of its 180B-parameter Darwin model that it says runs on a laptop with 8GB of VRAM and 32GB of RAM. Air-gapped teams still need a measured token rate on that laptop before they can price it against a server.
Reality
Ollama and llama.cpp added local endpoints within a week for Jev decision models, 144M-to-27B checkpoints that return a label and a probability. Routing calls can move off hosted models once teams check each checkpoint's licence and test it on their own labels.
Reality
Ollama 0.35.1 picks Gemma 4's prompt template from the model's name, so a copied or renamed 12B model loses four prompt tokens when thinking is off. Teams that save the library model under their own name can pin RENDERER gemma4-large in the Modelfile to keep the template that matched Google's.
Reality
llama-server b11430 reports logprob 0.0 for every speculatively decoded token, dragging one test's mean logprob from -0.48 to -0.0011. Nothing in the response or the server log flags the fill-ins, so evals and calibration built on those numbers go wrong quietly.
Reality
AMD published Gorgon Halo AI benchmarks days before Nvidia's expected RTX Spark launch, claiming a 1.1x to 32.2x lead over Intel's Core Ultra X9 388H. Until RTX Spark is measured, buyers weighing Gorgon Halo machines that cost upwards of $7,099 can compare the two only on memory.
Reality
Strata runs a 125B-class Qwen model at about 100 tok/s on one RTX 4090 because its sparse MoE activates only about 6B parameters per token. The design moves the hardware bill to system RAM, and the 100 tok/s figure holds mainly for code and structured output.
Reality
Ollama since 0.34.4 lets Gemma 4 skip the requested JSON schema, returning bare text with HTTP 200 in 8 of 24 test calls. Until the open fix ships in a release, structured output on local thinking models needs a shape check in the client.
Reality
Salvatore Sanfilippo's open-source ds4 engine runs a short list of large open-weight models locally by compressing their routed experts to about two bits. Even compressed, the supported builds need high-memory Macs or GPU systems that most people do not already own.
Reality
VIDRAFT says POCKET-Darwin-180B, a 111 GB 4-bit GGUF build of its 180B mixture-of-experts model, runs in llama.cpp on about $1,400 of consumer hardware. The accuracy evidence so far is one MMLU-Pro comparison that VIDRAFT reports itself.
Reality
llama.cpp merged a /v1/systemone endpoint on October 2 that returns typed answers with probabilities from five open model families running locally. Whether it can replace hosted classification calls depends on calibration that each team has to measure on its own data.
Perspective Coverage
3 publishersReality
DeepAgents runs in a four-file Docker Sandbox kit whose agent can reach only a local Model Runner on port 12434, with no cloud keys. The egress policy is careful work, and exact reproduction still rests on what PyPI serves when each sandbox is created.
Reality
Leadership · August 25, 2026 · Confirmed2 publishers
Security · August 25, 2026 · Confirmed2 publishers
Build · September 17, 2026 · Confirmed2 publishers