llama.cpp

llama.cpp delivers LLM and VLM inference in C/C++ with minimal setup and performance across a wide range of hardware. The project emphasizes a plain C/C++ implementation without dependencies and treats Apple silicon as a first-class target, optimizing for NEON, Accelerate, and Metal. It supports x86 architectures with AVX, AVX2, AVX512 and AMX, and RISC-V with several vector extensions. The tool is aimed at developers who want efficient local or cloud inference without external runtimes.

llama.cpp operates on top of the ggml library and provides backends and tools to build and run models. It includes a quick start with install scripts, Docker, and pre-built binaries, and offers model downloads from Hugging Face. The project also exposes commands to run a model directly or to launch an OpenAI-compatible API server, indicating an emphasis on practical deployment workflows.

The main capabilities include quantization for faster inference and smaller memory usage across multiple bit-widths (1.5-, 2-, 3-, 4-, 5-, 6-, and 8-bit), custom CUDA kernels for NVIDIA GPUs, and support for AMD GPUs via HIP and MUSA. It offers Vulkan and SYCL backend support, CPU+GPU hybrid inference, and multimodal tooling through a modular ggml ecosystem, highlighting flexibility for various hardware setups.

llama.cpp lists plans for running on Docker, Android builds, multi-GPU usage, and XCFrameworks, along with various integration and documentation options. It mentions domains like model hosting with llama cli and llama serve, and supports building from source, using prebuilt binaries, and installing through scripts, showing a breadth of platform and integration paths.

Use cases

• Local or cloud LLM/VLM inference with minimal setup