Inference Radar

TL;DR

Kimi K3 set the serving agenda: Moonshot’s new frontier MoE model pushed routing, prefix caching, long-context memory, and multi-backend support to the top of the queue for vLLM.1
MoE kernels became the shared battleground: SGLang, FlashInfer, TensorRT-LLM, vLLM, ROCm AITER, and LightLLM all moved MoE serving work forward.2
Local runtimes chased frontier-model behavior: llama.cpp, Ollama, oMLX, MLX Swift, and LocalAI focused on tool calling, speculative decoding, KV cache, and new model families.3
Edge stacks got more serious: Google LiteRT, ExecuTorch, MNN, Qualcomm AI Hub, ncnn, sherpa-onnx, and RunanywhereAI expanded on-device model support and hardware backends.4
Security and ops hardening cut across the stack: Ray, llamafile, LocalAI, LMDeploy, LiteLLM, and ktransformers fixed unsafe defaults, stale processes, SSRF paths, and deserialization risks.5

This Week in Inference

Moonshot AI’s Kimi K3 was the week’s main model event, a trillion-parameter-class MoE with a long-context target that stresses expert routing, prefix caching, KV memory, and disaggregated prefill/decode systems.6 Meituan also released LongCat, and Thinking Machines introduced Inkling, which adds another foundation-model entrant that will need routing, eval, and serving infrastructure even if its first path is API-first.7 Google’s Gemma 4 weight refresh also mattered because silent artifact changes can shift latency, tool calling, and vision behavior for teams that pin model names rather than exact files.8

The technique side stayed focused on low-bit serving and long-context execution. BsQAT pushed block-wise shared quantization for large LLMs, DisQ reported weight-only INT4 gains, and LBLLM pushed a W2A4 path with small training budgets BsQAT.9 The runtime pattern matched the papers: FlashInfer, SGLang, vLLM, TensorRT-LLM, ROCm AITER, OpenVINO, ONNX Runtime, and MNN all spent the week on FP4, FP8, INT4, KV compression, speculative decoding, or MoE routing paths FlashInfer.10

Hardware news kept the same shape. NVIDIA’s Rubin disclosures and Blackwell throughput claims framed inference as a rack-scale efficiency problem, while edge reports reminded teams that thermal throttling and memory limits still define mobile LLM UX NVIDIA coverage.11 Capital also kept flowing into inference infrastructure, with General Compute, SambaNova, and Together AI showing that investors now treat inference chips and serving capacity as core AI infrastructure General Compute.12

Deeper Dive

Everything below is for readers who want the full picture. Feel free to scroll.

Code Changes by Category

Cloud & Datacenter Serving

vLLM had one of the busiest weeks in the serving layer, with DeepSeek and DSv4 kernel work, FlashInfer MoE routing, speculative decoding fixes, Rust frontend request prep, multimodal Qwen and Cosmos fixes, and Gaudi plugin parity work vLLM.28 SGLang moved more kernels into its own package, added SM100 and SM120 acceleration paths, expanded AMD and Helios coverage, and kept tuning HiCache and disaggregated serving SGLang.29 TensorRT-LLM added multi-process serving, MTP decoding, KV compression, VisualGen CuTe kernels, Qwen3-VL media handling, and MiniMax sparse paths TensorRT-LLM.30

Ray Serve moved SGLang toward Ray-native execution through SGLang’s RayEngine, direct streaming, co-located routing, and per-proxy ingress scaling Ray.31 Dynamo shipped a stable feature release centered on KV-aware routing, branch-sharded KV indexing, topology-aware routing, EPP integration, and Kubernetes GPU deployment work Dynamo.32 LocalAI hardened distributed serving with model-scoped shutdown, port quarantine, worker discovery fixes, readiness semantics, and wrong-model request rejection LocalAI.33

LiteLLM had a very large proxy week, adding router correctness fixes, provider compatibility, MCP enterprise authorization, cost and cache accounting, Model Armor sanitization, guardrails, and admin UI test hardening LiteLLM.34 LMDeploy removed interactive session-continuity semantics in favor of stateless OpenAI-style inference while adding AgRs all-to-all MoE support and SSRF mitigation LMDeploy.35 Triton Inference Server focused on reproducible build presets and repo-local CI tooling rather than serving features Triton Server.36

Local LLM Runtimes

llama.cpp added DeepSeek V4 graph and tokenizer work, speculative decoding sidecar improvements, Laguna support, Qwen3-VL updates, CUDA virtual devices, Vulkan, WebGPU, SYCL, OpenCL, Hexagon, KleidiAI, and broad server UI fixes llama.cpp.37 Ollama added Laguna parser and renderer fixes, GLM tool-call parsing, MLX and llama.cpp updates, CUDA platform coverage, direct I/O for Linux GPU iGPUs, and a new agent skills system Ollama.38 llamafile shipped transcribefile, HTTPS support, restored Linux CPU sandboxing, and two curated llama.cpp syncs llamafile.39

oMLX shipped a fast Apple Silicon release train with native Laguna, Bonsai low-bit kernels, Nemotron-H MTP, GLM MTP quant loading, Apple M5 workarounds, and long-context scheduler fixes oMLX.21 ktransformers added Intel iGPU SYCL support for GPTQ INT4 MoE inference, improved RAWINT4 Kimi prefill, fixed AVX-VNNI checkpoint loading, and tightened an archived scheduler RPC bind address ktransformers.40 DS4 added GLM support, distributed Metal and CUDA inference, DSpark speculative decoding hooks, and ROCm paths DS4.41

Apple Silicon & MLX Ecosystem

Apple’s MLX stack focused on inference correctness and speed, including gemv_wide, SDPA corruption fixes, Metal resource reuse, zero-copy unified-memory CPU import, and quantized matmul fixes MLX.19 MLX Swift LM landed TurboQuant KV-cache compression, Gemma VLM preprocessing, Qwen VLM memory work, MTP speculative decoding, MoE internals, and tool-calling fixes MLX Swift LM.42 mlx-vlm added native text model coverage, Inkling support, DiffusionGemma warm starts, Qwen Omni and VL fixes, APC and TurboQuant cache compatibility, and structured decoding speedups mlx-vlm.43

mlx-audio and mlx-audio-swift cut realtime audio latency through VAD-gated STT, Granite Speech NAR optimization, incremental VoxtralRealtime frontend processing, quantized cache paths, and Qwen3-ASR fixes mlx-audio.44 FluidAudio fixed ASR tail loss, streaming diarization timestamp drift, iOS resource signing, stalled Hugging Face downloads, and Core ML compute-unit controls FluidAudio.45 osaurus shipped Mac-native agent and MLX/vMLX runtime fixes around Bonsai, Gemma paged cache, ZAYA cache behavior, Computer Use, Browser Use, TTS, MCP, and app polish osaurus.46

Mobile & Edge Frameworks

Google’s AI Edge stack moved across LiteRT, LiteRT-LM, XNNPACK, Gallery, samples, litert-torch, and MediaPipe, with work spanning QNN, OpenVINO, Apple Foundation Models, low-bit XNNPACK kernels, Google Tensor, WebGPU, PhotoTalk, and Hugging Face export LiteRT-LM.47 ExecuTorch added Qualcomm GenAI scaffolding, WebGPU matmul and attention work, Vulkan SDPA and quantized-linear improvements, Arm composable quantizer defaults, TOSA and Cortex-M fixes, and XNNPACK quantized Conv1d behavior ExecuTorch.48 Alibaba MNN prepared its next release while fixing OpenCL Adreno attention, Metal LLM prefill and decode kernels, Qwen quantization quality, Hexagon NPU direct programming, mmap failure handling, and Arm82 long-context stability MNN.49

Qualcomm AI Hub added SAM3, YAMNet, OWL-ViT, D-FINE, MobileFaceNet, BEVDet, VLM SpinQuant, Qwen3 eval alignment, and GenieX tool-call and Windows on Snapdragon fixes Qualcomm AI Hub Models.50 Tencent ncnn had light merged code but active work on RotaryEmbed correctness, interpolation scale semantics, Vulkan crashes, and multimodal model conversion ncnn.51 sherpa-onnx cut FireRedASR decoder memory and latency, fixed Node async TTS use-after-free, improved Java and Rust examples, and hardened TTS Unicode handling sherpa-onnx.52

Compilers, Runtimes & Graph Engines

Triton Lang added Rubin bring-up, Blackwell and TMEM stabilization, descriptor and TMA correctness fixes, AMD gfx1250 work, top-k serving latency fixes, and profiling improvements Triton.53 OpenXLA advanced GPU collectives, command buffers, ROCm Triton correctness, oneAPI and SYCL support, profiling chunking, IFRT deserialization, and HLO safety checks OpenXLA.54 TensorFlow mirrored much of that compiler work while also improving LiteRT and TensorFlow Lite Micro aliasing behavior TensorFlow.55

TVM and tvm-ffi focused on ABI compatibility, Python wrapper identity, TIRx cleanup, TensorRT BYOC, ONNX import, Vulkan SPIR-V interfaces, and deterministic tvmjs sampling TVM.56 ONNX Runtime added low-bit CPU kernels, QMoE NVFP4 CUDA work, WebGPU/WebNN SDPA and MatMul paths, ARM and RISC-V quantized kernels, and many correctness fixes ONNX Runtime.57 OpenVINO added ONNX compatibility, GPU fp8 KV-cache GQA, zero-copy cache loading, NPUW work, DFlash speculative decoding in GenAI, and NNCF INT2 and INT3 compression fixes OpenVINO.58

Models, Quantization & Optimization

Hugging Face Transformers added Cosmos 3 Edge, Qwen3-ASR, Qwen3 Omni tests, multimodal fixes, LoRA key-mapping fixes, compressed-tensor fixes, packed MoE fixes, DeepGEMM, and FP8 kernels Transformers.59 Diffusers removed Flax and JAX support, added Cosmos and Anima pipelines, added a schema-aware CLI, and improved LoRA lifecycle handling Diffusers.60 Optimum Intel added OpenVINO paths for LTX video, DFlash draft export, Qwen3 Omni Talker optimization, Gemma token-type support, and compression CLI coverage Optimum Intel.61

FlashInfer pushed Expert Parallelism, NIXL, all-to-all, CUTLASS MoE workspace, unified MoE, FP4, FP8, SVDQuant, Blackwell kernels, and autotuner fixes FlashInfer.10 ROCm AITER and ATOM worked across DeepSeek, GLM, MiniMax, MLA, FP8 KV cache, speculative decoding, RDNA paths, and release packaging AITER.62 FlashAttention fixed SM100 FP8 E4M3 accuracy and added paged-KV cache overflow checks FlashAttention.63

Other Notable Changes

DeepSpeed fixed ZeRO-3 activation checkpointing with frozen parameters, pipeline gradient scaling, Muon reduce-scatter, DeepCompile pass contracts, and inference config compatibility DeepSpeed.64 JAX released a major update and added faster approximate random samplers, TPU support, Mosaic GPU and Pallas fixes, XLA bumps, and LAPACK hardening JAX.65 Luminal added general-purpose autograd, a CUDA semantic-search overhaul, fused MoE decode work, and more PyTorch Dynamo import coverage Luminal.66

Community Pulse

Open WebUI had no landed code but saw heavy issue traffic around RAG knowledge visibility, OpenAI-compatible API boundaries, mobile chat history regressions, build memory, CORS configuration, and translation workflow Open WebUI.67 DeepSeek-V3 had no code movement but users filled the issue tracker with agent-memory, drift, persistent profile, feedback-loop, and session-resumption proposals DeepSeek-V3.68 FastChat saw maintenance proposals around LaTeX rendering, local image loading safety, rope-scaling config tolerance, older PyTorch OOM handling, and decoding cleanup FastChat.69

Exo had no merged code but active proposals around zenoh reconnect hardening, token-relay decode, heterogeneous layer placement, configurable timeouts, packaging, and Claude/OpenAI proxy compatibility exo.70 BentoML had no merged code but incoming fixes for CPU-only TransformersRunnable, custom pipeline validation, IO descriptor typing, YAML CLI validation, Windows bento imports, and CI dependencies BentoML.71 oobabooga saw proposed fixes for per-chat token statistics, portable launch flags, preset filename validation, streaming Markdown, Laguna support, oversized prompts, and unsupported NVFP4 UX text-generation-webui.72

Community Debates

SGLang maintainers closed several disaggregation and kernel proposals after successor work or better-scoped fixes appeared, including Inkling support, MORI RDMA token clamping, and a fused FlashInfer RoPE plus quant plus KV append path SGLang debate.73 The pattern shows a project that wants model-specific speedups, but only when they fit the broader kernel and overlap model.

vLLM closed hidden-state extraction, token-string-collision logprobs, speculative placeholder fixes, KV-connector cleanup, and DP tuning proposals after maintainers pushed for different API or runtime directions vLLM debate.74 The project’s issue volume shows demand for more introspection, but maintainers kept the serving API tight.

Open WebUI rejected forwarding chat_id into the OpenAI user field because the behavior would affect all OpenAI-compatible backends, and maintainers pointed users toward Filters for backend-specific behavior Open WebUI debate.75 That is the clearest boundary line this week between gateway convenience and compatibility discipline.

FlashAttention closed CuTe DSL dropout support after reviewers asked for stronger backward-mask tests and flagged correctness and efficiency concerns FlashAttention debate.76 The review shows how hard it is to add stochastic features to kernels that serve as infrastructure for many higher-level runtimes.

Triton Server reversed from centralized org-level pre-commit and PR tooling toward repo-local CI, closing shared-template proposals after deciding that each repository should carry its own workflow config Triton Server debate.77 The decision favors local reproducibility over central governance.

Worth Watching

Kimi K3 support will likely expose the next round of failure modes in MoE routing, long-context cache policy, speculative decoding acceptance, and disaggregated prefill/decode vLLM.78 NVFP4 and MXFP4 are moving from isolated kernels into runtime policy across FlashInfer, SGLang, vLLM, TensorRT-LLM, ROCm, and llama.cpp FlashInfer.79 Apple Silicon runtimes are converging on the same primitives as cloud servers, including KV quantization, speculative decoding, tool calling, VLM preprocessing, and model catalog management MLX Swift LM.20

Edge inference is gaining deployment shape through LiteRT, ExecuTorch, Qualcomm GenieX, MNN, sherpa-onnx, RunanywhereAI, and ncnn RunanywhereAI.80 Security work will stay visible because local servers now expose agent tools, file access, remote media fetches, model downloads, and cross-replica state LocalAI.81 GGUF remains the practical local artifact format, but quantization semantics, model templates, and backend kernels still fragment across runtimes llama.cpp.82

Major Releases

Dynamo shipped v1.3.0 plus two prereleases, focused on router buildout, standalone selection service, branch-sharded KV indexing, compressed-radix-tree hot paths, topology-aware routing, parser-layer tool-calling alignment, and Kubernetes GPU deployment work. The stable release makes Dynamo a stronger orchestration layer for KV-aware serving across vLLM, SGLang, and TensorRT-LLM backends..32

AMD Ryzen AI shipped GAIA v0.22.0, gaia-agent-email v0.5.0 prerelease, and Lemonade v11.5.0. The theme was local agent infrastructure: email-agent workflows, Agent Hub packaging, Lemonade Router, server-side jobs, MCP client hosting, HTTPS CLI support, and tighter CORS defaults. Lemonade release.83

Apache TVM shipped tvm-ffi v0.1.13-rc1, a release candidate centered on stable Python object identity, ObjectPtr support, Optional and Variant ABI layout stabilization, and dtype alias expansion. TVM consumed the compatibility work through wrapper lifetime and device-definition fixes..84

BerriAI shipped a large LiteLLM release and backport train spanning v1.90.5 through v1.95.0-dev.1. The dominant theme was proxy hardening: router correctness, MCP enterprise authorization, provider compatibility, cost and cache accounting, guardrails, signed Docker verification, and admin UI reliability. Latest release.34

Blaizzy shipped mlx-vlm v0.6.5 and v0.6.6. The releases expanded DeepSeek, LFM2, Qwen, Gemma, and native text-model coverage while fixing APC and TurboQuant warm-cache behavior and adding structured-decoding speedups. Latest release.43

Dao-AILab shipped fa4-v4.0.0.beta23 for FlashAttention. The prerelease packaged the paged-KV block-table bounds check and the SM100 FP8 E4M3 max-offset accuracy fix, two changes that reduce memory-corruption risk and FP8 numerical error..85

FlashInfer shipped v0.6.15, v0.6.15.post1, and daily nightlies. The release line put Expert Parallelism in the default install, expanded TRTLLM-GEN MoE, broadened Blackwell support, added Video Sparse Attention, and tightened FP8 all-reduce fusion and autotuning reliability..16

ggml published 43 llama.cpp build releases, with b10091 as the latest in-window tag. The release train covered DeepSeek V4 fixes, Laguna support, speculative sidecars, CUDA quantized GET_ROWS, WebGPU, Vulkan, Hexagon, KleidiAI, SYCL, and package lookup fixes. Latest release.86

Hugging Face shipped Transformers v5.14.1. The patch release focused on Inkling integration regressions, including assisted generation with EncoderDecoderCache and StaticCache plus SDPA prefill when position bias is used..87

JAX shipped jax-v0.11.0. The release added experimental hijax custom-derivative docs and helpers, top-level jax.custom_remat, and a broad set of runtime, sampler, TPU, Mosaic GPU, and XLA updates landed around it..65

jundot shipped oMLX v0.5.2.dev1, v0.5.2rc1, v0.5.2rc2, v0.5.2, and v0.5.3. The train focused on Apple Silicon serving stability, Bonsai low-bit kernels, Nemotron-H MTP, Laguna support, GLM MTP quant loading, M5 kernel workarounds, and long-context cache fixes. Latest release.21

Mozilla AI shipped llamafile 0.10.4. The release introduced transcribefile, added HTTPS support, restored pledge and SECCOMP sandboxing for Linux CPU serving and CLI modes, and synced llama.cpp with Vulkan and ROCm acceleration improvements..39

NVIDIA shipped TensorRT-LLM v1.3.0rc22. The prerelease continued the dense serving cycle around multi-process serve, speculative decoding, KV compression, VisualGen kernels, multimodal handling, and known multi-GPU and FP8 issues..88

Ollama shipped v0.32.1 and withdrew v0.32.2. The useful release improved Gemma 4 tool calling and multi-turn reasoning, fixed an MLX cache leak, sped cache snapshots, improved MLX load-timeout handling, and added agent working-directory context..89

osaurus-ai shipped 0.22.5 through 0.22.8. The releases focused on Mac-native agent reliability, Bonsai and Gemma paged-cache paths, AppleScript and Computer Use finalization, plugin hardening, OpenAI-compatible TTS, pinned chats, multi-select actions, and telemetry fixes. Latest release.46

Qualcomm shipped ai-hub-apps v0.33.0 and GenieX/nexa-sdk v0.3.16. ai-hub-apps added fetch --device support and updated CLI dependencies, while GenieX shipped a Microsoft-signed HTP catalog for llama.cpp on Windows on Snapdragon. GenieX release.24

Ray shipped Ray 2.56.1. The patch release focused on Ray Data to_pandas regressions, Arrow-backed pandas opt-out support, tensor-column crash fixes, and memory-monitor improvements for system-slice pressure..90

ROCm shipped AITER v0.1.16.post5, AITER v0.1.18, and AMDMIGraphX rocm-7.14. AITER focused on wheel recovery and long-context 64-bit MLA support, while MIGraphX rolled up ONNX resize parsing, attention struct ordering, MLIR updates, docs-core updates, and naming cleanup. AITER release.91

Kimi K3 Shoves Routing Into The Hot Path