oMLX

oMLX is a macOS-native inference server designed to run locally on Apple Silicon Macs. It serves as a backend for AI models, enabling developers to run and manage large language models without relying on remote cloud infrastructure. The tool includes a web dashboard and a native macOS app to monitor and control the server from the system tray. It emphasizes persistence of cache blocks to disk to improve responsiveness during repeated context use.

The system architecture combines a two-tier caching approach with a native interface. Cache blocks are saved to disk in safetensors format, and frequently used blocks stay in RAM while cold blocks move to SSD. This setup helps restore previously seen prefixes quickly across requests and restarts without recomputing.

It supports multiple model types and can load LLM, VLM, embedding, and reranker models simultaneously. The solution uses an eviction policy to manage memory when needed and allows browsing and downloading models directly through the admin dashboard. It also provides a drop-in API compatible with OpenAI and Anthropic endpoints, enabling integration with various clients.

The product outlines its platform requirements and integration points, noting it works with Apple Silicon and macOS 15+, provides a native menu bar app, a web dashboard, and a Hugging Face cache compatibility. It also mentions support for tool calling formats and a range of model families, with a focus on local, offline-friendly inference.

Use cases

• Developers run local AI inference on Apple Silicon Macs to avoid remote latency and dependencies.
• Coding agents maintain and reuse large context efficiently by persisting KV cache.