Magnitude

Magnitude is an inference engine for developers and enthusiasts who run open large language models locally and want the fastest possible speed on their own hardware, whether a Mac, an NVIDIA or AMD GPU, an Intel chip, or just a CPU. It is a faster alternative to tools like Ollama and llama.cpp.

When a user downloads a model, Magnitude spends up to a minute tuning that model specifically for the chip it is running on, trying different ways to split up the math, such as how many GPU threads work together and how far ahead to load data, and keeps the fastest combination that does not change the model's output. It installs as a desktop app and a CLI, and connects with one click to agents a developer already uses, such as Pi, OpenCode, Hermes, or Codex.

Magnitude reports being more than 2x faster than Ollama on average across popular local models on Mac and NVIDIA hardware. Other capabilities include a built-in speculative decoding drafter for each model, quantized KV memory that halves memory use per agent and frees it when agents stop, and shared prefix caches across concurrent sessions to prevent slowdown.

Magnitude is free, private, and open source under the Apache-2.0 license, with no token costs and nothing leaving the user's machine. It has more than 7.7k GitHub stars and 555 forks, and can also run as a remote server for other computers or inside a Docker container.

Use cases

• Running open models locally without sending data off the machine
• Running Magnitude as a shared server for other computers