BitNet

BitNet (bitnet.cpp) is the official inference framework for 1-bit large language models such as BitNet b1.58. It offers optimized kernels for fast, lossless inference of 1.58-bit models on CPU and GPU, with NPU support planned. It is aimed at developers who want to run large language models locally with much lower compute and energy requirements than full-precision models.

Developers clone the repository, install Python and build dependencies, then download a model such as BitNet-b1.58-2B-4T from Hugging Face and run setup_env.py to quantize and prepare it with either the i2_s or tl1 kernel type. Inference then runs through run_inference.py, which accepts a model path, a prompt, and options such as thread count, context size, and temperature, with an optional conversational mode for chat.

bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs and 2.37x to 6.17x on x86 CPUs compared with full-precision inference, while cutting energy consumption by up to 70.0% on ARM and 82.2% on x86. It can run a 100B-parameter BitNet b1.58 model on a single CPU at speeds comparable to human reading, and offers an official GPU inference kernel plus BitNet-embedding-0.6B and BitNet-embedding-270M embedding models for CPU-based retrieval tasks.

BitNet requires Python 3.10 or later, CMake 3.22 or later, and Clang 18 or later, with additional Visual Studio 2022 components needed on Windows. It is released under the MIT license and free to use. The project uses existing 1-bit LLMs already published on Hugging Face to demonstrate bitnet.cpp's inference capabilities rather than training new models itself.

Use cases

• Running large language models locally on consumer CPU hardware
• Real-time multilingual speech recognition on CPU