Tracked Repositories

HuggingFace Transformers — state-of-the-art NLP/ML model library (~140K stars)

HuggingFace Diffusers — diffusion model inference & training (Stable Diffusion, Flux, etc.)

Minimalist Rust ML framework for inference — targets browser WASM and GPU, zero Python dependency

HuggingFace TGI — LLM serving (archived March 2026, read-only)

Industry-standard deep learning framework with XLA compilation backend

TensorFlow Serving — high-performance gRPC/REST serving for TF models (multi-version, canary, batching)

TensorFlow Lite for microcontrollers and embedded devices

User-friendly local LLM runner built on llama.cpp (~167K stars)

High-performance LLM inference in C/C++ (CPU + GPU)

High-performance Whisper speech recognition in C/C++

Self-hosted ChatGPT alternative with built-in RAG, offline-capable (~104K stars)

Primary ML framework; torch.compile + AOTInductor for production inference optimization

PyTorch's portable execution framework for on-device inference

TorchServe — production PyTorch model serving (archived August 2025)

Reference inference code for DeepSeek-V3 (671B MoE); includes FP8 training framework

Most widely adopted open-source LLM serving engine; PagedAttention, continuous batching

vLLM community plugin for Intel Gaudi accelerators

Cross-platform ML pipeline framework (vision, audio, NLP)

AI Edge model gallery

LiteRT for language model inference

Official Gemma model cookbook — recipes, fine-tuning, deployment guides

Sample apps using MediaPipe

Google's Lite Runtime (successor to TensorFlow Lite)

Highly optimized neural network operators library (ARM, x86, WASM)

Model visualization and exploration tool

LiteRT integration with PyTorch

Sample code for LiteRT

Quantization tooling for AI Edge models

Web samples for MediaPipe

Sample models for AI Edge

Command-line tooling for LiteRT

Evaluation tooling for AI Edge models

Google AI Edge documentation site

Desktop AI app + SDK for running LLMs locally (~73K stars)

Array framework for ML on Apple silicon (Python)

Example models and applications using MLX

Reverse-engineered Apple Neural Engine (ANE) — hardware ops, memory layout, firmware interactions

LLM inference and fine-tuning with MLX

Tools for converting & running models with Core ML

Example apps using MLX Swift

Swift bindings for MLX

LLM inference in Swift via MLX

Efficient data loading for MLX

C bindings for MLX

Unified OpenAI-compatible proxy for 100+ LLM providers (vLLM, Ollama, Bedrock, Azure, etc.)

Gradio web UI for LLMs — multi-backend (llama.cpp, ExLlamaV2, transformers) (~43K stars)

Free, open-source OpenAI drop-in replacement — runs locally, no GPU required (~36K stars)

Run LLMs distributed across heterogeneous devices (Mac, iPhone, etc.)

Distributed AI compute engine; Ray Serve handles online and async batch inference (~39K stars)

Microsoft DeepSpeed — distributed training and inference (ZeRO, MII, FastGen)

Open Neural Network Exchange format specification

Microsoft's cross-platform, high-performance ONNX inference engine

LLM serving framework and home of Chatbot Arena (~37K stars)

NVIDIA's optimized LLM inference library (GPU)

NVIDIA's high-performance deep learning inference SDK (GPU)

CUDA C++ templates for high-performance matrix-multiply (GEMM) and convolution kernels

C++ LLM/VLM inference runtime for Jetson and NVIDIA edge devices

Composable NumPy transformations (JIT, grad, vmap) compiled via XLA to GPUs and TPUs — primary DeepMind research/production runtime

MLC's universal LLM deployment engine (multi-backend)

Tile-based ML language and compiler

Community on-device LLM project

vLLM-style inference on Apple silicon via MLX

High-throughput LLM/VLM serving with RadixAttention and structured generation

High-performance neural network inference for mobile (Android/iOS)

Tencent Neural Network — mobile and edge inference

Modular Platform monorepo — MAX inference server/framework + Mojo programming language for portable, high-performance AI on CPUs and GPUs

Single-file LLM executables via Cosmopolitan Libc — zero install, all platforms (~21K stars)

Official FlashAttention — fast, memory-efficient exact attention (FA-2/FA-3) kernels for GPUs

Python-like GPU kernel language used by vLLM FlashAttention and PyTorch inductor

High-performance LLM inference in web browsers via WebGPU

CPU-GPU hybrid inference; runs DeepSeek 671B on 14GB VRAM + 382GB DRAM with massive speedup over llama.cpp

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Alibaba's neural network inference framework for mobile & edge

Apache TVM ML compiler — auto-tunes models for any hardware target

Apache TVM Foreign Function Interface for deep learning compilation

Audio models (TTS, ASR) with MLX

Vision-language models on Apple silicon via MLX

Swift audio inference using MLX

Text embedding models with MLX

Video model inference with MLX

ONNX-based runtime for ASR, TTS, VAD, and keyword spotting

Intel's toolkit for optimizing & deploying deep learning on Intel hardware

Neural Network Compression Framework — quantization, pruning, sparsity for OpenVINO

OpenVINO GenAI — generative AI layer with speculative decoding & KV-cache opt

RunAnywhere SDKs for on-device inference deployment

RunAnywhere CLI tool

Intel IPEX-LLM — local LLM acceleration on Intel hardware (archived Jan 2026, read-only)

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity

Official minimal inference library for all Mistral models (7B, Mixtral, Pixtral)

NVIDIA Triton — production multi-model inference server (HTTP/gRPC, multi-backend)

On-device LLM/VLM SDK for Snapdragon NPU, GPU, and CPU (formerly Nexa AI)

State-of-the-art ML models optimized for Qualcomm Snapdragon NPU/DSP/QNN deployment

Sample apps and tutorials for deploying models on Qualcomm hardware (TFLite, ONNX, QNN)

DNN inference library & tutorials for NVIDIA Jetson

Unified serving framework: real-time APIs, task queues, batching, multi-model chains

High-throughput LLM serving with TurboMind engine (C++/CUDA)

Datacenter-scale distributed inference serving framework (Rust + Python, disaggregated prefill/decode, engine-agnostic)

Lightweight inference engine for mobile & embedded from PaddlePaddle

On-device Whisper inference for Apple platforms (Swift)

Python tooling for WhisperKit model optimization

On-device AI benchmarking framework

Swift playground for ArgMax SDK

Native macOS AI agent harness in Swift — any model, persistent memory, autonomous execution, MCP server, MLX + Apple Neural Engine, fully offline

High-performance GPU kernel library for LLM serving — attention, sampling, and KV-cache primitives

Cactus core edge inference framework

React Native bindings for Cactus

Kotlin/Android bindings for Cactus

Flutter bindings for Cactus

Demo chat app using Cactus

High-performance EXL2-quantized inference for consumer NVIDIA GPUs

Fast C++ inference for Transformer models; INT8/INT16 CPU quantization, multi-platform

Compiler for JAX, TF, PyTorch targeting GPU, TPU, and CPU from a unified IR

Lightweight, high-throughput Python-based LLM inference and serving framework

Multi-LoRA inference server — serve thousands of fine-tuned adapters on a single GPU

Examples, tutorials and apps for Liquid AI LFM + LEAP SDK

Speech-to-Speech audio models by Liquid AI

Minimal fine-tuning repo for LFM2, fully open-source

Example apps for LeapSDK

Liquid AI documentation

Rust-based deep learning compiler with a small static graph IR for fast, portable inference (CUDA, Metal, CPU)

On-device audio inference framework

Fluid Inference core runtime

Rust text processing library for inference

Mirai's on-device inference runtime

Mirai's LLaMA-based on-device model

Multimodal LLM inference framework for mobile & edge

ARM Neural Network SDK for ARM & Mali devices

AI Tensor Engine for ROCm — centralized repo for high-perf AI operators on AMD Instinct GPUs

AMD's graph inference engine for MI-series GPUs

ROCm fork of FlashAttention with Composable Kernel (CK) and Triton backends

AiTer Optimized Model — lightweight vLLM-like server built on AITER kernels for ROCm

NimbleEdge's deliteAI on-device inference framework

NimbleEdge fork of ExecuTorch with edge optimizations

A simple, fast and robust program-aware agentic inference system

Picovoice's on-device LLM inference engine

MLange sample applications

MLange extension library

iOS framework for MLange

MLange SDK documentation

iOS extension framework for MLange