TL;DR
- MoE serving became the center lane: vLLM, SGLang, TensorRT-LLM, ROCm, FlashInfer, and LightLLM all pushed model routing, expert kernels, KV movement, or disaggregated serving paths.1
- Apple-local inference accelerated: Ollama, Apple MLX, oMLX, MNN, Osaurus, and mlx-vlm all shipped or merged work around MLX, Metal, Laguna, long context, and local agent loops.2
- Edge GenAI got more complete: MediaPipe, LiteRT, LiteRT-LM, ExecuTorch, MNN, sherpa-onnx, Runanywhere, and Qualcomm AI Hub all expanded mobile, web, NPU, and packaged-device workflows.3
- The kernel layer was busy: FlashInfer, FlashAttention, Triton, TileLang, AITER, CUTLASS, and ONNX Runtime advanced FP4, FP8, NVFP4, MXFP8, MoE, attention, and Blackwell or AMD paths.4
- The OpenAI-compatible layer hardened: LiteLLM, Open WebUI, LocalAI, Dynamo, Ray Serve, and LightLLM worked on routing, auth, tool calls, streaming, billing, observability, and gateway safety.5
This Week in Inference
The market signal was not one giant open-weight model launch. Anthropic released Claude Opus 5 with a user-facing effort control, xAI added Grok STT, and Moreh showed GLM-family distributed inference across AMD MI300X systems, but the open-source code story sat one layer lower, engines racing to make Kimi, DeepSeek, Qwen, Gemma, Laguna, MiniMax, and GLM-family models run well across datacenter and local stacks. vLLM added Kimi K3 and Qwen-family paths while shipping a broad serving release, and SGLang introduced DSpark speculative decoding while pushing Kimi/KDA and disaggregated serving work in the same cycle (vLLM).1
The strongest technical pattern was disaggregation. vLLM, SGLang, Dynamo, TensorRT-LLM, LightLLM, Ray Serve, FlashInfer, and ROCm all touched some mix of KV transfer, prefill/decode split, expert routing, NIXL, Mooncake, radix cache, HiCache, or MoE kernels. SGLang’s release framed speculative decoding as a serving feature, while FlashInfer and AITER filled in the lower-level FP4, FP8, MXFP8, NVFP4, and expert-parallel pieces that make those server paths viable (SGLang).6
The same stack is moving downward. Ollama, MLX, oMLX, MNN, Osaurus, Runanywhere, Qualcomm AI Hub, LiteRT, ExecuTorch, sherpa-onnx, and TensorRT-Edge-LLM all shipped local or edge-facing improvements tied to the same concerns as the cloud engines: long context, KV cache, quantization, speech, tool calls, and model packaging. MediaPipe’s first stable release and ONNX Runtime’s new release made the week feel less like “cloud versus edge” and more like one inference supply chain with different placement targets (MediaPipe).3
Top Stories
vLLM and SGLang turn day-zero MoE serving into a race
vLLM shipped a major release centered on new model support, DeepSeek performance work, multimodal serving, structured output fixes, speculative decoding, Mamba and hybrid models, and KV/cache infrastructure (vLLM).1 SGLang answered with a release led by DSpark speculative decoding, Kimi/KDA work, disaggregated serving, Radix/HiCache refactors, ROCm and Ascend support, and diffusion/VLM fixes (SGLang).6 The point is not that one server won the week. The point is that high-end inference engines now compete on how fast they absorb new model architectures, route KV state, and keep MoE serving stable.
Apple-local inference got a full-stack push
Ollama focused on MLX and Apple GPU support, adding Laguna support and fixing MLX Metal quality issues for NVFP4 output (Ollama).2 Apple’s MLX ecosystem added CUDA and Metal kernel fixes, Poolside Laguna support, DeepSeek and Hunyuan support in Swift, Gemma VLM video input, and Qwen decode work (Apple MLX).7 oMLX tightened long-context memory admission, Laguna DFlash paths, speculative/MTP behavior, and macOS chat UX in one prerelease (oMLX).8 The Mac stack is no longer just “run a quantized model locally”; it is becoming its own serving platform with prompt caches, long-context guards, tool flows, and model-specific acceleration.
Edge GenAI moved from demos to runtime plumbing
Google shipped MediaPipe’s first stable release while LiteRT, LiteRT-LM, litert-torch, samples, Gallery, and XNNPACK all moved on-device GenAI forward across WebGPU, model export, runtime correctness, low-bit CPU paths, and app UX (MediaPipe).3 Meta’s ExecuTorch added backend-agnostic off-graph KV cache support, LLM correctness fixes, Stable Diffusion wrappers, and more backend coverage across Vulkan, WebGPU, XNNPACK, Arm, Qualcomm, and NXP (ExecuTorch).9 Qualcomm, Runanywhere, MNN, and sherpa-onnx expanded the same edge surface through QHexRT, Hexagon, SPM, speech, TTS, and packaging work (Qualcomm AI Hub Models).10
Kernel teams are building the MoE substrate
FlashInfer spent the week on unified MoE quantization, expert-parallel reliability, XQA, MLA, speculative decode, FP4, FP8, NVFP4, and distributed collectives (FlashInfer).4 FlashAttention advanced FA-4 varlen backward, FP8 numerical behavior, CUDA graph capture, and stable ABI support for downstream servers such as vLLM and SGLang (FlashAttention).11 ROCm’s AITER and ATOM pushed Kimi-K3, DeepSeek, GLM, SGLang, and RAPIDserve paths into AMD’s inference stack, showing that AMD support now means model-serving integration rather than raw kernel availability (AITER).12
The gateway and admin plane kept getting more serious
LiteLLM hardened provider adapters, routing, billing, SSO, MCP/OAuth, auto-routing, Docker signing, and operational telemetry for enterprise OpenAI-compatible gateways (LiteLLM).5 Open WebUI shipped a major UI rebuild while fixing auth loops, SSRF-safe loaders, tool-call memory behavior, access control, terminal behavior, and accessibility (Open WebUI).13 LocalAI added a vLLM C++ backend, 3D generation, voice/TTS work, Valkey Search vectors, distributed model loading fixes, and security hardening around inline reward code (LocalAI).14
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM had the broadest cloud-serving week: Kimi K3, Qwen-family support, Inkling, MiniMax, diarized transcription, structured-output fixes, speculative decoding, Mamba and hybrid-cache work, ROCm/XPU/CPU updates, and a Gaudi plugin that tracked upstream compatibility closely (vLLM).1 SGLang was just as intense on architecture: DSpark speculative decoding, Kimi/KDA, Mamba, Radix and HiCache internals, disaggregated serving, NIXL/Mooncake, ROCm, Ascend, diffusion, and VLM work all landed together (SGLang).6
NVIDIA’s TensorRT-LLM work centered on MoE, KV cache, disaggregation, model support, sampling, and CI, while TensorRT-Edge-LLM released a Jetson and edge-focused package with Gemma, Qwen, Whisper, Parakeet, Nemotron-Omni, speculative decoding, logprobs, and CuTe DSL kernels (TensorRT-Edge-LLM).15 Triton Inference Server aligned to its new release, updated container and compatibility docs, fixed protobuf and gRPC build breakage, and expanded QA for TensorRT CUDA graphs, lifecycle shutdowns, Dynamo launch behavior, and logging callbacks (Triton Server).16
Ray Serve moved tokenization and routing decisions into the LLM ingress layer, added arm64 Ray LLM images, updated vLLM CI, added an SGLang Grafana dashboard, and hardened shutdown, proxy, and scheduler behavior (Ray).17 Dynamo worked on disaggregated-serving reliability through GPU CI, vCluster and Grove hardening, KV routing and replay, OpenAI-compatible protocol behavior, and Kubernetes/operator readiness (Dynamo).18
Local LLM Runtimes
llama.cpp expanded model and multimodal coverage across MiniMax-M3, Nemotron Omni, MiMo audio, GLM Vision, Nanbeige, Laguna, Qwen Omni fixes, speculative decoding, MCP stdio, and a wide backend sweep across CUDA, HIP, SYCL, Metal, Vulkan, OpenCL, WebGPU, PowerPC, and Hexagon (llama.cpp).19 Ollama focused on Apple GPU and MLX support, Laguna, Qwen MoE correctness, draft-model output-head quantization, memory residency, speculative diagnostics, and agent/TUI skill workflows (Ollama).2
LocalAI added a vLLM C++ backend with continuous batching and paged KV cache, plus 3D generation, voice controls, Valkey Search vectors, distributed model loading, proxy caching, and security hardening (LocalAI).14 Mozilla’s llamafile updated its embedded llama.cpp, refreshed CUDA patches, clarified release binaries, and fixed transcribefile packaging for future release artifacts (llamafile).20
Apple Silicon & MLX Ecosystem
Apple’s MLX projects focused on CUDA MXFP8 quantization, CUDA GEMM/conv fixes, Metal kernel cleanup, Poolside Laguna support, DeepSeek and Hunyuan support in Swift, Gemma VLM video input, and Qwen decode performance (MLX).7 mlx-vlm and mlx-audio expanded native embeddings, Mage and Kimi model support, AWQ and mixed-precision compressed tensors, streaming audio fixes, server-thread ownership, and Swift audio checkpoint compatibility (mlx-vlm).21
oMLX focused on long-context admission, memory pressure, Laguna DFlash acceleration, speculative/MTP fixes, VLM correctness, macOS chat UX, localization, and benchmark honesty (oMLX).8 Osaurus shipped a rapid release train around MLX/vMLX cache correctness, SSD prefix reuse, autonomous agent loops, provider compatibility, channel integrations, and Chinese localization (Osaurus).22
Mobile & Edge Frameworks
Google’s AI Edge stack had one of the deepest edge weeks: MediaPipe shipped its first stable release, LiteRT added backend and WebGPU fixes, LiteRT-LM advanced speech, TTS, ASR, embeddings, GPU/OpenCL/WebGPU/NPU fixes, and XNNPACK landed low-bit and robustness work (MediaPipe).3 ExecuTorch added off-graph KV cache infrastructure, LLM correctness fixes, Stable Diffusion wrappers, Android backend options, and backend work across Vulkan, WebGPU, XNNPACK, Arm, Qualcomm, and NXP (ExecuTorch).9
MNN shipped Hexagon NPU support, Metal LLM acceleration, Transformer fused paths, QLoRA-aware paths, and backend fixes across OpenCL, QNN, CUDA, Metal, Vulkan, TensorRT, and Hexagon (MNN).23 sherpa-onnx added Swift Package Manager support, ASR and TTS fixes, Windows ARM64 wheels,.NET Android runtime packaging, ONNX Runtime version APIs, and multi-platform examples (sherpa-onnx).24
Compilers, Runtimes & Graph Engines
ONNX Runtime shipped a new release with ONNX and protobuf updates, CUDA EP dependency changes, FP4/FP8 block-quantized ops, CUDA decode work, WebGPU ops, MLAS kernels, and packaging fixes (ONNX Runtime).25 ONNX tightened checker behavior, exception-disabled builds, ASan coverage, Scan shape handling, and opened a broader quantization type-system discussion for K-quants, NF4, MXFP4, BitNet, and IQ formats (ONNX).26
OpenVINO advanced Gemma, Qwen, LTX video quality, GPU correctness, GGUF recurrent/dynamic-token support, ONNX frontend coverage, snippets, CPU platform plumbing, and GenAI benchmarking ahead of its next release train (OpenVINO).27 TVM and TVM-FFI focused on TIRx correctness, CUDA/Blackwell lowering, Relax/ONNX frontend fixes, Rust object ergonomics, and wheel/ABI reliability (TVM).28
Models, Quantization & Optimization
Hugging Face added Granite-SWA, SKT A.X, Gemma internals, Krea2, Cosmos3, Qwen-image LoRA work, Candle Metal buffer APIs, and Optimum Intel validation for Gemma and Qwen-family exports (Transformers).29 FlashInfer, AITER, Triton, TileLang, FlashAttention, ONNX Runtime, and vLLM all pushed low-bit serving paths, with repeated focus on FP4, FP8, MXFP8, NVFP4, compressed tensors, and MoE kernels (FlashInfer).4
AMD’s ktransformers release brought end-to-end fine-tuning for large sparse MoE models through KT-managed BF16 CPU-side buffers, Full-FT, LoRA, and hybrid training (ktransformers).30 DeepSpeed improved DeepCompile and ZeRO-3 lifecycle behavior, tensor-parallel inference, LM-head sharding, scheduler validation, MoE support, and CI safety (DeepSpeed).31
Other Notable Changes
LiteLLM hardened enterprise gateway behavior across provider adapters, routing, credential isolation, spend accounting, SSO, MCP/OAuth, auto-router behavior, Docker signing, telemetry, and guardrails (LiteLLM).5 Open WebUI rebuilt its UI while landing auth, SSRF, file-upload, tool-call, terminal, accessibility, and access-control fixes (Open WebUI).13
Community Pulse
The week’s raw activity was huge: the dataset covered 158 repos, 96 repos with git activity, 4,807 commits, 3,699 merged PRs, 1,599 new issues, and 84 new releases. vLLM, SGLang, Open WebUI, LiteLLM, Ollama, llama.cpp, Ray, TensorFlow, OpenXLA, Google AI Edge, Meta, ROCm, and FlashInfer were among the highest-volume areas by commits, PRs, releases, or issue flow (vLLM).1
Community energy clustered around model enablement and hard bug reports. Kimi K3, DeepSeek, GLM, Qwen, Gemma, Laguna, MiniMax, and Nanbeige threads showed up across servers, local runtimes, MLX projects, ROCm, and edge frameworks (SGLang Kimi roadmap).32 Security and reliability reports also mattered: unauthenticated pickle over ZMQ appeared in both LMDeploy and LightLLM issue traffic, while Open WebUI, LiteLLM, LocalAI, Triton, TensorFlow, XNNPACK, ONNX, and BentoML all had security or correctness hardening work in view (LMDeploy security issue).33
Community Debates
vLLM closed or redirected several large architecture proposals, including AFD disaggregation, EAGLE CUDA Graph speculative decoding, TurboQuant/HIGGS online compression, heterogeneous attention geometry, and KV-offload headroom policy, with maintainers pushing toward clearer ownership, benchmark evidence, and unified policy design (vLLM AFD proposal).34 SGLang had similar design convergence around Kimi-Linear heterogeneous transfer, decode-offload KV handling, KDA numerical behavior, masked-scale lifetime fixes, and stale monitoring proposals (SGLang debate).35
Open WebUI rejected unconditional orjson replacement despite claimed CPU gains, then landed an opt-in path instead, a good example of deployment safety winning over a broad performance switch (Open WebUI debate).36 LiteLLM closed guardrail and router work where reviewers found metadata bypasses, update-path concerns, Redis expiry risks, and memory-bound session capture questions (LiteLLM debate).37
Edge and compiler projects showed stricter scope control. ExecuTorch maintainers pushed an in-tree Nordic NPU backend out of tree until backend-hosting rules mature, Triton maintainers rejected dynamic-shape checks because Triton tensors are statically shaped, and TVM rejected predicated gemm_async after benchmarks did not justify the API surface (ExecuTorch debate).38
Worth Watching
Kimi K3 is now the best proxy for whether engines can handle new MoE architectures fast. vLLM, SGLang, ROCm AITER and ATOM, FlashInfer, MLX projects, oMLX, LocalAI, and community Apple-cluster threads all pointed at Kimi-related support or blockers this week (SGLang Kimi roadmap).32
The browser and web edge may be next. web-llm users asked for speculative decoding in WebGPU, MediaPipe and LiteRT added WebGPU-facing work, ONNX Runtime added WebGPU ops, and ExecuTorch kept WebGPU in its backend sweep (web-llm request).39
MoE disaggregation is moving from research blog topic to product surface. The pieces now span routers, KV transfer, expert-parallel collectives, speculative decode, FP4/FP8 kernels, and hardware roadmap language, and the open-source projects are starting to expose those pieces as deployable controls rather than hidden internals (FlashInfer).4
Major Releases
This is the canonical release reference for the week.
vLLM shipped v0.26.0, a large release with 411 commits from 212 contributors. The dominant theme was fast model enablement plus production serving correctness, led by Inkling support, DeepSeek-V4 performance work, Kimi K3, Qwen-family paths, multimodal transcription, structured outputs, speculative decoding, and KV/cache changes..1
SGLang shipped v0.5.16, headlined by DSpark speculative decoding and a wide sweep across Kimi/KDA, Mamba, RadixCache, HiCache, disaggregated serving, ROCm, Ascend, VLM, and diffusion paths. The release also highlighted 574 PRs from 169 contributors, underscoring SGLang’s current velocity..6
Ollama shipped v0.32.3 through v0.32.5, focused on Laguna support, Apple MLX paths, MLX Metal quality fixes, Qwen MoE decoding, draft-model output-head quantization, stalled downloads, and integration fixes. The most important change was making Laguna and MLX more usable on Apple GPUs. Latest release.2
ggml shipped a rapid llama.cpp binary train from b10092 through b10182. The releases covered MiniMax-M3 vision, Nanbeige4.2, MiMo audio, GLM-5.2 NextN/MTP, Laguna, Metal FWHT, CUDA NVFP4, Mamba-2 CUDA work, OpenCL/WebGPU fixes, SYCL fast paths, and common suppress-token handling. Latest release.19
Open WebUI shipped v0.11.0 with a major visual redesign across chat, admin, menus, settings, typography, spacing, and conversation layout. The release also absorbed fixes around uploads, tool calls, SSRF hardening, auth loops, access control, terminal behavior, accessibility, and sandbox execution..13
LiteLLM shipped v1.94.0 plus a prerelease, with release notes emphasizing signed Docker image verification via cosign. The broader release week focused on enterprise gateway correctness, provider adapters, routing isolation, spend accounting, SSO, MCP/OAuth, auto-router lifecycle, observability, and guardrails..5
Google AI Edge shipped MediaPipe v1.0.0. The release included build and dependency changes, Gecko Text Embedder support, WebGPU service changes, Apple rules migration, Abseil updates, and synchronized sample work across mobile and web..3
Microsoft shipped ONNX Runtime v1.28.0. The release upgraded ONNX and protobuf, changed CUDA EP dependencies, made cuDNN and cuFFT optional runtime dependencies, removed nvrtc as required runtime dependency, and sat alongside heavy FP4/FP8, CUDA decode, WebGPU, and MLAS work..25
NVIDIA shipped TensorRT-Edge-LLM v0.9.1. The release focused on Gemma 4 multimodal support, DFlash, paged KV cache, logprobs and logit bias, CuTe DSL kernels and artifacts, export and quantization updates, and runtime stability..15
Triton Inference Server shipped v2.71.0 for the 26.07 NGC container. The release week updated compatibility docs and defaults, repaired gRPC/protobuf build fallout, improved SDK Docker reliability, and expanded TensorRT CUDA graph, lifecycle, logging, Dynamo, and batcher QA..16
Alibaba MNN shipped 3.6.1. The release added a Qualcomm Hexagon NPU backend, Transformer C4 fused paths across CPU/Metal/OpenCL/CUDA, Metal LLM prefill and decode improvements, MNN-aware QLoRA, and stability fixes around mmap allocation, KV-cache decode indexing, and Arm82 LinearAttention precision..23
FlashInfer shipped v0.6.16 release candidates plus nightlies. The release train focused on unified MoE quantization, TRT-LLM MXFP4 paths, expert-parallel fault tolerance, XQA and MLA support, FP4/FP8 fixes, SM12x and SM100 work, and packaging/JIT cleanup. Latest release candidate.4
FlashAttention shipped fa4-v4.0.0.beta24. The prerelease bundled FA-4 varlen backward score-mod support, stable ABI dropout-header cleanup, CUDA graph combine-kernel scheduling, and FP8 numerical changes..11
ROCm shipped AITER v0.1.19. The release centered on Kimi-K3 FlyDSL SiTUv2 MoE kernels, strided grouped-topk, tuned gfx950 and gfx1250 configs, and ROCm 7.x wheels for gfx942 and gfx950..12
AMD Ryzen AI shipped FastFlowLM v0.9.46, Lemonade v11.5.1, and RyzenAI-SW v1.8.0. The release train aligned FastFlowLM’s transition to ROCm, Lemonade’s FLM backend and routing work, and RyzenAI-SW’s expanded model, TTS, embeddings, translation, Stable Diffusion, and Quark docs. FastFlowLM release.40
DeepSpeed shipped v0.19.3. The patch release covered version updates, fp16 dynamic loss-scaling validation, merge-queue DCO workflow, CI cancellation cleanup, and a ZeRO-3 per-parameter dtype fix, while the week’s merged work pushed DeepCompile, AutoTP, scheduler, MoE, NVMe, and CI safety forward..31
kvcache-ai shipped ktransformers v0.6.4. The release delivered high-performance end-to-end Full-Parameter and LoRA fine-tuning for large sparse MoE models using KT-managed BF16 CPU-side buffers, hybrid CPU-GPU training, and updated SFT kernels..30
Blaizzy shipped mlx-vlm v0.6.7 and v0.6.8 plus mlx-audio v0.4.6. The dominant themes were native model coverage, MLX VLM embeddings, AWQ and compressed-tensors support, Laguna and Mage models, streaming audio reliability, and audio-loading performance. mlx-vlm release.21
oMLX shipped v0.5.4.dev1. The prerelease focused on Mac usable-context benchmarking, predictable long-context memory admission, MiMo V2.5 text support, Laguna DFlash acceleration, web chat overhaul, quantization fixes, and integration polish..8
Osaurus shipped 0.22.9 through 0.22.12. The release train hardened MLX/vMLX cache behavior, SSD prefix reuse, autonomous agent loops, channel integrations, provider compatibility, proxy handling, keychain reliability, and Chinese localization. Latest release.22
Qualcomm shipped ai-hub-models v0.59.0 and GenieX v0.3.17. The releases covered AI Hub CLI/export defaults, BEVDet on NPU, YAMNet packaging, Nomic embedding assets, and a Microsoft-signed HTP catalog for Windows on Snapdragon. ai-hub-models release.10
RunanywhereAI shipped v0.20.11 and v0.20.12 of runanywhere-sdks. The releases published Android/Kotlin, QHexRT, QAIRT, SwiftPM, MLX resources, and Apple assets while the repo added Python SDK bindings and aligned SDK defaults across Android, Kotlin, Swift, Flutter, React Native, Web, Electron, Python, and CLI. Latest release.41
try-mirai shipped lalamo v0.14.1 and uzu 0.5.14. The releases focused on server FSDP/TP, LongRoPE, stable SiLU, Weaver speculative decoding, A8W8 quantization, Metal packaging, tool calling, and runtime cleanup. lalamo release.42
ai-dynamo shipped v1.4.0-kimi-k3-dev.1. The snapshot is an experimental, non-QA-gated Kimi-K3 development build for Dynamo’s vLLM backend, and the wider week focused on GPU CI, Kubernetes/operator readiness, KV routing/replay, frontend protocol behavior, and disaggregated serving reliability..18