TL;DR
- Sparse agent serving got real: NVIDIA Nemotron 3.5 Lightning pushed always-on MoE agent serving into mainstream runtimes as Ollama and edge stacks added support.1
- Datacenter engines chased the same workloads: vLLM and SGLang focused on Kimi, DeepSeek, DSpark, MoE, FP4, and production scheduling paths.2
- Local and Apple stacks moved from hobby to cluster: Ollama, llama.cpp, MLX, oMLX, vllm-mlx, and Osaurus all advanced model support, tool calling, cache behavior, and Mac serving reliability through oMLX.3
- Edge inference kept getting more complete: Google LiteRT-LM, Meta ExecuTorch, Qualcomm GenieX, sherpa-onnx, MNN, and UbiquitousLearning mllm all pushed mobile or on-device LLM, VLM, ASR, and TTS workflows through LiteRT-LM.4
- Kernel work set the agenda: FlashInfer, CUTLASS, Triton, TVM, OpenXLA, ROCm AITER, and TensorRT-LLM all worked on FP4, NVFP4, MoE, Blackwell, ROCm, and synchronization paths through FlashInfer.5
This Week in Inference
NVIDIA Nemotron 3.5 Lightning was the cleanest new model signal in the window: Ollama describes it as a 30B-class MoE model with roughly 3B active parameters for always-on agent harnesses, which matches the broader shift toward sparse active compute rather than smaller dense models alone Ollama.1 Muse Glimmer also became a cross-stack support wave, with Hugging Face adding it to Transformers, Ollama serving it through MLX paths, and local Apple runtimes picking it up across MLX, oMLX, Osaurus, and Uzu Transformers.6 Kimi, DeepSeek, Qwen, Gemma, GLM, MiniMax, and Nemotron drove most serving work, with vLLM, SGLang, TensorRT-LLM, LMDeploy, LightLLM, and Dynamo all tuning around those model families SGLang.7
The main technical direction was clear: FP4 and NVFP4 kernels, speculative decoding, and KV-cache policy moved from special cases into core serving machinery. FlashInfer shipped production MoE expert-parallel work and Blackwell FP4 paths, while CUTLASS shipped a CuTe DSL bugfix release that feeds the same kernel stack FlashInfer.5 vLLM added Kimi, DeepSeek, DSpark, DeepGEMM, compressed checkpoints, and shared-expert sharding, while SGLang pushed DSpark, diffusion/video serving, native MXFP4 serving, DCP, KDA-aware prefix caching, and HiCache vLLM.2 The same pattern showed up below the cloud layer: llama.cpp added model-loader hardening, speculative backend sampling, ROCm, Vulkan, Metal, SYCL, and Pocket-TTS work, while WhisperKit added incremental long-audio loading for Apple platforms llama.cpp.8
Hardware and infrastructure news came through software, not chip launches. Google shipped LiteRT-LM C API prebuilts for native app integration, Meta shipped ExecuTorch with broader Cortex-M, Arm, Vulkan, Qualcomm, and wheel support, and Qualcomm shipped GenieX with AI Hub model catalogue and OpenAI-compatible serving fixes LiteRT-LM.4 NVIDIA pushed edge LLM serving with TensorRT Edge-LLM, including ONNX-less engine building, KV reuse, video serving, Qwen3-TTS voice features, and CuTe DSL FMHA migration TensorRT Edge-LLM.9 The supplied market briefing did not surface a major funding or acquisition event inside the week; the visible industry movement came through open GitHub work on hybrid deployment, signed images, release trains, and edge packaging Ray.10
Top Stories
vLLM and SGLang turn MoE, DSpark, and Kimi into production tracks
vLLM shipped a large serving release centered on Kimi, Python and Rust frontends, AttnRes kernels, DeepGEMM, compressed checkpoints, DSpark AR fusion, and shared-expert sharding vLLM.2 SGLang’s release notes show the same direction from another angle: Kimi day-one support, native MXFP4 serving, DCP, DSpark, KDA-aware prefix caching, HiCache, LoRA, and quantization all moved together SGLang.7 The two projects now compete less on “OpenAI-compatible server” basics and more on which stack can keep sparse, long-context, multi-modal, and speculative workloads stable under production load vLLM-Gaudi.11
FlashInfer and CUTLASS set the low-precision kernel pace
FlashInfer shipped production-ready MoE expert parallelism, SM12x fused-MoE FP4 accuracy updates, unified MoE MXFP4 support, and vLLM decode coverage FlashInfer.5 CUTLASS shipped CuTe DSL fixes for TMA bulk copy, fp32-to-f8 conversion, grouped GEMM DGLU compilation, JIT overhead, kernel naming, and import speed CUTLASS.12 These libraries sit under the serving layer, so their fixes ripple into vLLM, TensorRT-LLM, SGLang, FlashAttention, and edge runtimes that need FP4 or NVFP4 to behave like a speed path rather than a compression checkbox TensorRT-LLM.13
Google and Meta make edge LLMs feel like a platform
Google shipped LiteRT-LM with C API shared-library prebuilts for native app integration and language bindings LiteRT-LM.4 The surrounding Google AI Edge repos added multimodal Gallery import, MediaPipe LiteRT-LM executor paths, Model Explorer .litertlm conversion support, and XNNPACK performance and safety work Gallery.14 Meta shipped ExecuTorch with expanded Cortex-M, Arm backend, Vulkan desktop GPU, wheel, and Qualcomm support, making the edge runtime story broader than mobile demos alone ExecuTorch.15
Ollama, llama.cpp, and MLX push local agents into the model-support fast lane
Ollama shipped a rapid run of releases for Muse Glimmer, Nemotron, MLX performance, DFlash, image-input support, OpenAI-compatible Responses work, web search, and agent tool edits Ollama.16 llama.cpp kept pace with model-loader hardening, Muse Glimmer, Granite-Switch, Nemotron, DFlash, Gemma conversion fixes, Pocket-TTS, multimodal server state, and backend acceleration across Vulkan, CUDA, OpenCL, Arm, ROCm, WebGPU, Metal, and SYCL llama.cpp.8 Apple’s MLX ecosystem added distributed work, quantized kernels, Swift LLM support, cache policies, and Muse Glimmer paths through MLX core, mlx-swift-lm, mlx-vlm, and oMLX oMLX.3
Audio, speech, and VLMs became first-class edge workloads
argmax shipped WhisperKit incremental file loading for long audio, faster Qwen3-TTS, public SpeakerKit centroid embeddings, and Apple-platform fixes WhisperKit.17 sherpa-onnx shipped a broad platform rollup covering Flutter WebAssembly, ONNX Runtime updates, CUDA builds, ASR numerical fixes, TTS emotion IDs, speech enhancement, and many language bindings sherpa-onnx.18 Alibaba MNN, FluidAudio, UbiquitousLearning mllm, and Qualcomm GenieX all worked on on-device VLM, ASR, TTS, or multimodal paths, which makes edge inference a speech and vision problem as much as a text problem GenieX.19
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM’s week centered on Kimi, DeepSeek, DSpark, DeepGEMM, MoE, ROCm, XPU, AMD Zen, Gaudi alignment, and release security, with the Gaudi plugin also adding GPT-OSS MXFP4, GLM fallback paths, and security signing vLLM.2 SGLang matched that volume with Kimi K3, DeepSeek-V4, DSpark, diffusion/video serving, MiniMax, Cosmos, native MXFP4, DCP, HiCache, LoRA, AMD HIP, and ROCm CI work SGLang.7
NVIDIA’s serving stack split across TensorRT-LLM and TensorRT Edge-LLM: the edge release added Nemotron, Cosmos, DiffusionGemma, DSpark, ONNX-less engine building, KV reuse, and video serving, while TensorRT-LLM continued Kimi, disaggregated serving, KVCacheManagerV2, speculative decoding, and sparse-attention work TensorRT Edge-LLM.9 LMDeploy added Hy3, Intern-S2-Mobius meta-MoE, Hopper blocked-FP8 GEMM, TurboMind VLM support, OpenAI reasoning-token accounting, and a JSON replacement for pickle in disaggregated-serving request decoding LMDeploy.20 LightLLM shipped RL serving with verl, disaggregated ViT serving, hybrid cache for linear-attention models, cache-aware PD serving, and MTP speculative decoding LightLLM.21
Dynamo had a large production-hardening week across HTTP shutdown, tool-call normalization, Anthropic tool IDs, token-budget rejection, vLLM multimodal sidecars, SGLang serving, elastic EP, router policy, Kubernetes, encrypted dispatch, and large-model recipes for Qwen and Nemotron Dynamo.22 Ray shipped a release centered on Ray Data V2 defaults and shuffle scalability, while Serve added tracing, configurable backpressure responses, proxy crash fixes, and batch-inference ergonomics for vLLM and SGLang Ray.10 LiteLLM pushed router affinity, tag filtering, virtual routing groups, cooldown policy, proxy logging, SSE keepalive, provider translation, PTU flat-cost accounting, and admin UI work LiteLLM.23
Local LLM Runtimes
Ollama’s release train focused on Muse Glimmer, Nemotron, MLX, DFlash, NVFP4 prefill speed, OpenAI-compatible Responses, web search, tool edits, image input, and duplicate-digest OCI safety Ollama.16 llama.cpp added loader hardening for malformed GGUF and metadata, Muse Glimmer, Granite-Switch, Nemotron, Gemma conversion fixes, DFlash metadata, speculative backend sampling, Pocket-TTS, and backend acceleration across Vulkan, CUDA, ROCm, OpenCL, Arm, WebGPU, Metal, and SYCL llama.cpp.8 LocalAI added NeMo-Speech.cpp, Qwen3-TTS, MiniMax video and audio generation through vllm.cpp, vLLM engine options, shared WebRTC UDP support, parallel Hugging Face downloads, reversible PII redaction, and gallery fallback fixes LocalAI.24
vllm-mlx shipped Apple Silicon serving fixes around scheduler behavior, streaming finish reasons, MTP, media owner-thread routing, native mlx-lm chunked prefill, memory-budget reporting, and model compatibility vllm-mlx.25 DS4 advanced DSpark direct verifier-state commits, Metal DFlash verification, ROCm speculative decoding, M5 Metal decode fusions, MXFP4 and Q2 paths, long-context fixes, and server disconnect tests DS4.26 GPT4All did not merge code, but contributors opened HTTPS/TLS verification and local API CORS hardening proposals GPT4All.27
Apple Silicon & MLX Ecosystem
Apple’s MLX work spanned distributed JACCL diagnostics, scatter-reduce all-reduce, quantized matmul fixes, faster large 1-D dot products, searchsorted, numerical stability, autograd, linalg, and API cleanup MLX.28 mlx-swift-lm added Muse Glimmer and ATEM tools, Harmony parsing, KV-cache policy, prompt-cache persistence, MTP sliding-window overlays, prefill speedups, TurboQuant, model fixes, and SwiftPM compatibility mlx-swift-lm.29 Blaizzy’s mlx-audio removed hard mlx-lm dependency paths while adding Irodori-TTS, Nemotron VoiceChat, Qwen3-TTS dtype fixes, Higgs Audio, and streaming cleanup mlx-audio.30
oMLX had the clearest Apple cluster story: it added dashboard-driven tensor and pipeline parallel serving across Macs, reported Qwen tensor-parallel speedups, added MiniMax rank support, hardened planner and SSH behavior, and began Metal/CUDA pooled serving experiments oMLX.3 Osaurus shipped three releases around local inference, DSV4 Flash, Muse Glimmer, Nemotron, tool-calling fixes, sandbox and MCP work, parallel Range downloads, and macOS stability Osaurus.31 WhisperKit added incremental audio loading, Qwen3-TTS speedups, public SpeakerKit centroid embeddings, and prompt, timestamp, transcription, and cache fixes WhisperKit.17
Mobile & Edge Frameworks
Google AI Edge had one of the week’s broadest edge pushes: LiteRT-LM shipped C API shared libraries, LiteRT added accelerator and runtime plumbing, MediaPipe added LiteRT-LM executor paths, Gallery added multimodal chat and Hugging Face LiteRT-LM import, and XNNPACK added AVX512FP16, YNNPACK, WASM, and safety work LiteRT-LM.4 Meta ExecuTorch shipped expanded Cortex-M, Arm, Vulkan, Qualcomm, wheel, and GenAI support, while core PyTorch opened its release branch and continued Inductor, CUDA, SDPA, XPU, and MPS work ExecuTorch.15 Qualcomm’s GenieX and AI Hub work added Gemma 4 recipes, Qwen3 support, AI Hub catalogue listing, OpenAI-compatible completions, reasoning-content splitting, streaming compliance, QDC coverage, and app deployment samples GenieX.19
Alibaba MNN added Qwen3 TTS, ASR, VL embedding, HunyuanOCR, InternVL OpenCL vision, DFlash lm_head sharing, transformer fusion, SME2 and NEON decode, Metal chunk prefill, Hexagon int4 and int8 paths, and safer QNN export MNN.32 UbiquitousLearning mllm added Qwen3.5 0.8B single-image, multi-image, and short-video CPU inference, then extended the same multimodal path to the 4B model mllm.33 Tencent ncnn added Vulkan extension discovery, shader macro refactoring, fp16sa shader fixes, AVX-512 BF16 float2bfloat, and HunyuanOCR community deployment threads ncnn.34
sherpa-onnx shipped platform-wide ASR/TTS/runtime updates across Flutter WebAssembly, Android, iOS, macOS, Java, Rust, Python, Node.js, HarmonyOS, Tauri, ONNX Runtime, CUDA, numerical stability, VITS emotion IDs, and DPDFNet speech enhancement sherpa-onnx.18 FluidAudio fixed Kokoro TTS crashes on Apple OS betas, timestamp-aware ASR duplicate-token removal, audio frame-count clamping, and vocabulary docs FluidAudio.35 RunAnywhereAI aligned eight SDK surfaces, moved consumer apps into standalone iOS, Android, Web, Electron, Flutter, and React Native repos, added QHexRT, Snapdragon Windows ARM64, Apple NeuRT, C++ metrics, and app CI gates RunAnywhere SDKs.42
Compilers, Runtimes & Graph Engines
TVM shipped a compiler-heavy release after CUDA/PTX modernization, Relax shared-KV attention, Gemma-oriented cache work, WebGPU and Metal fixes, ONNX PRelu support, and NVRTC architecture normalization TVM.43 Triton focused on synchronization modeling, race analysis, membar precision, Blackwell SM12x enablement, AMD gfx1250 work, histogram dtype policy, and CI tooling Triton.44 OpenXLA advanced backend-agnostic collectives FFI, ragged all-to-all, cuBLASLt correctness, CPU Gemma HLO benchmarks, ROCm TheRock migration, oneAPI SYCL FFI, and parser hardening OpenXLA.45
TensorFlow mirrored much of that XLA work while adding crash-to-exception hardening, TFLite StridedSlice overflow fixes, oneDNN batch matmul safety, XNNPACK delegate support for dynamically quantized fully connected ops, and tflite-micro DECODE operators for compressed-model runtime decoding TensorFlow.46 ONNX Runtime shipped a release that deprecated WebGL/JSEP in onnxruntime-web in favor of native WebGPU EP, while Olive added fused-MoE RTN quantization and ONNX added custom opaque types and shape-inference fixes ONNX Runtime.47 OpenVINO added Gemma shared-KV NPUW paths, FLUX.2 Klein NPU support, low-precision GPU fixes, INT4 compressed FullyConnected shared weights, Snippets AArch64 GEMM repacking, and GenAI model-benchmark updates OpenVINO.48
Models, Quantization & Optimization
Hugging Face Transformers shipped Muse Glimmer support, Optimum Intel added OpenVINO export and inference for Muse Glimmer and Qwen-Image, Diffusers added Wan-Animate and LTX work, and Candle fixed Metal and CUDA correctness paths Transformers.6 FlashAttention focused on FA4 and CuTe dynamic-shape correctness, SM100 varlen, block-sparse and SplitKV deadlock fixes, scheduler first-tile preservation, and JIT compile-key stability FlashAttention.49 Intel neural-compressor merged an AutoRound NVFP4 vLLM plugin with dense and MoE routing into native vLLM NVFP4 kernels Neural Compressor.50
ROCm’s AITER and ATOM combined kernel and serving work across GFX950 and GFX1250 MoE, MXFP4, FP8, Gluon, sparse MLA, stable Top-K, GLM, Kimi, DeepSeek, MTP, persistent MLA decode, LMCache, Mooncake, PyNCCL, and CI gating AITER.51 FlashInfer’s MoE and Blackwell work lined up with vLLM and TensorRT-LLM needs for FP4, NVFP4, sparse attention, paged KV, and all-reduce fusion FlashInfer.5 Modular MAX added DSpark Gemma serving paths, structured-output support, FP8 KV cache in NVFP4 recipes, hybrid JengaBlockManager support, expert-parallel dispatch, AMD MXFP4 and MXFP8 split-K fused epilogues, and Mojo post-release cleanup Modular.52
Other Notable Changes
DeepSpeed shipped patch releases for DeepCompile, ZeRO/offload, native pin memory, AutoEP checkpoint conversion, Ulysses checkpoint rank selection, ZeRO++ edge cases, PyTorch build compatibility, and op-builder import safety DeepSpeed.53 Triton Inference Server rewrote its LLM onboarding toward the PyTorch backend and LLM API workflow, then aligned builds for the next container with ONNX Runtime, DCGM, OpenVINO, and Python updates Triton Server.54 Cactus merged a custom transpiler branch and continued Stable Diffusion pipeline work, while Luminal added compile-search early stopping, progress UI, and attention export fixes Cactus.55
Community Pulse
Open WebUI had one of the largest community weeks with DNS resolver regression fixes, Tavily loader repair, split-origin avatar fixes, queued-message release behavior, metadata parsing, localization, and active debates around hybrid Qdrant search and access control Open WebUI.56 DeepSeek’s repo had no code, but users filed detailed reports around DeepSeek-V4 Flash tool-call stalls, long-context repetition, CJK token corruption, reasoning-content loss, and backend parity between TileLang, vLLM, SGLang, API, and web deployments DeepSeek-V3.57 ktransformers had a light code week but heavy SFT and FSDP2 issue traffic around expert-LoRA saves, DTensor mixing, fused expert loading, batch sizing, and observability ktransformers.58
WebLLM had no merges, but users reported a suspected shape-cache lifetime regression causing WebGPU hangs and a structured-output structural_tag generation hang WebLLM.59 exo had no merged code, but open work targeted KV prefix-cache drift after chained extensions and dashboard accessibility, while users reported macOS memory growth and local-network permission visibility problems exo.60 CTranslate2 users asked for PyPI sdists, modern CUDA CMake, Blackwell-safe architecture parsing, CPU dispatch fallbacks, $ORIGIN rpaths, and non-GCC toolchain support CTranslate2.61
Community Debates
Maintainers across runtimes kept rejecting broad performance bundles in favor of smaller, testable changes. llama.cpp closed MoE expert caching and offload proposals after users reported speedups, with maintainers asking for an RFC and staged PRs because the changes affected backend complexity, build scope, and prompt-processing tradeoffs llama.cpp.62 SGLang closed diffusion sleep/wake offload, session cache, TeaCache, and TurboQuant KV proposals because maintainers wanted stronger decomposition, backend abstraction, correctness tests, or better long-context performance evidence SGLang.63 vLLM split a large Blackwell backend proposal into smaller backend PRs after users supported the direction but reviewers pushed for scoped integration vLLM.64
Security and protocol debates were sharper than usual. LightLLM merged an out-of-bounds Phi-3 rotary fix but left open discussion around replacing pickle-based multinode startup handshakes with validated UTF-8/IP paths LightLLM.65 LMDeploy replaced one pickle-based disaggregated-serving request path with JSON and Pydantic, while a separate proposal to reject pickled update_weights payloads by default closed without visible discussion LMDeploy.20 GPT4All closed a local API wildcard-CORS hardening PR without a recorded maintainer rationale, even though the contributor argued that arbitrary websites could call a local server when it is enabled GPT4All.66
Several projects also clarified product boundaries. Ollama closed a custom MLX grammar sampler and pointed toward XGrammar integration rather than maintaining a separate PDA sampler Ollama.67 WebLLM, llamafile, FastChat, MLC-LLM, Triton Server, and BentoML all saw self-closed or closed docs PRs that tried to add generic OpenAI-compatible hosted gateway notes, with authors or maintainers deciding those notes were off-message for local, project-specific, or Triton-specific docs WebLLM.68 Open WebUI closed otherwise substantive PRs when they missed CLA, branch, or contribution-policy requirements, which shows how process gates now shape feature flow as much as technical review Open WebUI.69
Worth Watching
The next pressure point is persistent and programmatic KV-cache control. Dynamo has issues around persistent KV events, programmatic KV hints, and replay-backed DGDR search, while FlashInfer is asking where persistent KV for long-running local agents breaks first in practice Dynamo.70 vLLM, SGLang, TensorRT-LLM, oMLX, Ollama, exo, and vllm-mlx all had cache correctness, prefix reuse, rotating-cache, offload, or speculation issues this week vllm-mlx.71
The second thread is structured output under pressure. Ollama, LMDeploy, LiteLLM, Open WebUI, WebLLM, Osaurus, Uzu, and TensorRT-LLM all touched tool parsing, reasoning streams, JSON schema, grammar decoding, namespace tools, or streaming parser behavior LiteLLM.72 The industry keeps adding agent loops before parsers have settled, and that mismatch now shows up as truncation, leaked reasoning, tool-call loss, or stuck generation TensorRT-LLM.73
The third thread is that low-bit speed depends on native kernels. FlashInfer, CUTLASS, AITER, Triton, TensorRT-LLM, vLLM, SGLang, MNN, Uzu, and Intel neural-compressor all worked on FP4, NVFP4, MXFP4, INT4, or BF16 fast paths, which means the compression story is now inseparable from backend ownership Intel neural-compressor.50
Major Releases
vLLM shipped v0.27.0 and v0.27.1 this week, centered on Kimi K3, Python/Rust frontends, AttnRes kernels, DeepGEMM, compressed-tensors checkpoints, DSpark AR fusion, shared-expert sharding, and a patch for quantized DSpark Markov-head support. This was one of the week’s core datacenter serving releases..74
SGLang shipped v0.5.17, with release notes citing 582 PRs from 194 contributors and a focus on Kimi K3 day-one support, KDA linear attention, native MXFP4 serving, DCP, DSpark, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache, LoRA, and quantization. The release marks SGLang’s move into broader multimodal and speculative serving..7
Ollama shipped four versions from v0.32.7 to v0.32.10, focused on Muse Glimmer, Nemotron 3.5 Lightning, MLX availability, DFlash, faster NVFP4 prefill, repeat-penalty defaults, and OCI duplicate-digest safety. The cadence shows how fast local runtimes now absorb new model families. Latest release.16
FlashInfer shipped v0.6.17 plus several post and nightly builds, focused on MoE expert parallelism, SM12x fused-MoE FP4 accuracy, unified MoE MXFP4 support, and vLLM decode coverage. The project also repaired Python import and release-line issues through post releases..5
NVIDIA shipped CUTLASS v4.6.2, TensorRT Edge-LLM v0.10.0, and TensorRT-LLM v1.3.0rc24. The combined theme was CuTe DSL kernel infrastructure, edge LLM model support, ONNX-less engine building, KV reuse, video serving, and known-issue surfacing for complex CUDA graph and KV paths. TensorRT Edge-LLM release notes.9
Google AI Edge shipped LiteRT-LM v0.16.0 and Gallery 1.0.18. LiteRT-LM added first versioned C API shared-library prebuilts for supported platforms, while Gallery added multimodal chat and Hugging Face LiteRT-LM model-card URL import. LiteRT-LM release notes.4
Meta shipped ExecuTorch v1.4.0, highlighting expanded Cortex-M support, Arm backend coverage, Vulkan desktop GPU support, wheel work, and Qualcomm coverage. Core PyTorch also opened its next release branch, but ExecuTorch was the shipped inference artifact..15
Apache TVM shipped tvm v0.26.0 and tvm-ffi v0.1.13-post3. TVM’s release followed CUDA/PTX modernization, Relax shared-KV work, frontend fixes, and PyPI wheel publishing changes, while tvm-ffi added Python 3.14 wheel fixes. TVM release notes.43
ONNX Runtime shipped v1.29.0, deprecating WebGL/JSEP in onnxruntime-web in favor of native WebGPU EP and adding POSIX telemetry availability for telemetry-enabled builds. The release sat alongside heavy GenAI, WebGPU, CUDA, and security hardening work..47
Modular shipped MAX 26.5 / Mojo 1.0.0, moving GPU programming APIs out of the Mojo standard library into the new top-level max package. The release also framed Mojo layout and accelerator APIs as part of the MAX accelerator library..52
Ray shipped Ray 2.57.0, with Ray Data’s DataSourceV2 default and Hash Shuffle V2 as the headline changes. Serve and Data LLM work during the week added tracing, backpressure controls, vLLM/SGLang import deferral, multipart file-upload support, and control-plane hardening..10
LightLLM shipped v1.2.0, focused on RL serving with verl, disaggregated ViT serving, hybrid cache for linear-attention models, multi-level cache with cache-aware PD, MTP speculative decoding, API behavior, and long-running stability. The release also included PD scheduling and MoE serving work..21
LocalAI shipped v4.8.1 and v4.8.2, focused on backend and gallery reliability, terminal-agent docs, malformed GGUF metadata containment, release asset recovery, and dependency bumps. The week also added speech, TTS, vLLM options, reversible PII redaction, and gallery fallback work. Latest release.24
sherpa-onnx shipped v1.13.5 and a Flutter release, with a broad package rollout across mobile, desktop, web, language bindings, ONNX Runtime updates, CUDA builds, ASR numerical fixes, TTS emotion IDs, and WebAssembly examples. The release had one of the widest platform matrices of the week..18
WhisperKit / TTSKit / SpeakerKit shipped v1.1.0 through argmax’s Swift repo. The release added incremental long-audio loading, Qwen3-TTS speedups, public speaker centroid embeddings, and fixes for prompts, Chinese word timestamps, transcription options, TTS chunk boundaries, and model paths..17
Hugging Face Transformers shipped v5.15.0, adding Meta Muse Glimmer and related export support. The same week, Optimum Intel added OpenVINO export and inference paths for Muse Glimmer and Qwen-Image..6
Blaizzy shipped mlx-audio v0.4.8 and mlx-vlm v0.6.12 to v0.6.13. The dominant themes were audio model coverage, removal of hard mlx-lm dependency paths, APC prefix reuse, final-token stream flushing, Qwen structured-output fixes, and VLM cache correctness. mlx-vlm release notes.75
oMLX shipped v0.5.8.dev1 through v0.5.8.dev3, focused on Ling, Jina reranking, cache storage fixes, Muse Glimmer, DFlash speculative decoding, DeepSeek prefill speedups, bounded GDN SSD sidecars, and tool-call parsing. The release train also introduced distributed Mac cluster serving. Latest release.3
Osaurus shipped 0.22.18 through 0.22.20, focused on DSV4 behavior, MCP indexing, macOS crash fixes, cache-boundary capture, parallel downloads, Muse Glimmer runtime, reasoning validation, and vision-path fixes. The releases show fast local-agent hardening around native macOS inference. Latest release.31
Qualcomm GenieX shipped v0.3.19, with Docker action updates, release checklist docs, host binding hints, VLM text-length restoration, and HTP file-count docs. The surrounding week added AI Hub catalogue listing, Gemma 4 recipes, Qwen3 support, OpenAI-compatible completions, and reasoning-content splitting..19
RunAnywhereAI shipped v0.20.13 through v0.20.17, centered on Apple Neural Engine packaging, Swift NeuRT, XCFramework checksums, SwiftPM resolution, Electron QHexRT, and eight-SDK API alignment. The releases also moved consumer apps to standalone iOS, Android, Web, Electron, Flutter, and React Native repos and remote package consumption.36 Latest release.42
DeepSpeed shipped v0.19.4 and v0.19.5, focused on ZeRO, DeepCompile, op-builder safety, ZeRO++ fixes, workflow updates, and release bookkeeping. The week’s code also improved offload, checkpoint conversion, pin memory, and PyTorch build readiness. Latest release.53
Dao-AILab FlashAttention shipped fa4-v4.0.0.beta26, covering SM90 block-sparse backward waits, learnable sink backward support, FA4 scheduler first-tile preservation, and layer-norm cleanup. The release followed CuTe dynamic-shape, SM100, SplitKV, and scheduler fixes..49
AMD / ROCm FastFlowLM shipped v1.0.0 and v1.0.1 after moving FastFlowLM into the ROCm organization. The releases focused on project transfer, installer cleanup, MSI references, and SmolVLA benchmark results for Ryzen AI systems. Latest release.76
try-mirai shipped lalamo v0.15.0 and uzu 0.5.15. Lalamo added Uzu KV export, Weaver and DFlash changes, and quantized PLE enablement, while Uzu aligned runtime metadata after adding Muse Glimmer, quantized PLE, Weaver, DFlash, and Metal kernel work. lalamo release notes.77