FastChat

FastChat is an open platform for training, serving, and evaluating large language model based chatbots, maintained by LMSYS. It is built for researchers and developers who want to run, compare, and fine-tune open chat models such as Vicuna, and it powers Chatbot Arena, a site where users compare responses from more than 70 large language models.

A user installs FastChat with pip or from source, then downloads model weights such as Vicuna automatically from Hugging Face when running a chat command. Serving with the web interface requires a controller to coordinate the system, one or more model workers that host the models, and a Gradio web server that users interact with directly.

FastChat supports a wide range of models including Llama 2, Vicuna, Alpaca, ChatGLM, Falcon, GPT4ALL, Guanaco, OpenAssistant, StableLM, and WizardLM, and it can serve API-based models from OpenAI, Anthropic, Gemini, and Mistral alongside locally hosted ones. It offers a command-line interface, a distributed multi-model serving system with OpenAI-compatible RESTful APIs, 8-bit compression and CPU offloading for memory-constrained machines, and training and evaluation code including the MT-Bench benchmark.

FastChat is released under the Apache-2.0 license and runs on single or multiple GPUs, CPU-only machines, Apple Silicon or AMD GPUs through the Metal backend, Intel XPU hardware, and Ascend NPUs. Running Vicuna-7B needs about 14GB of GPU memory and Vicuna-13B about 28GB, though 8-bit compression can roughly halve those requirements, and quantization formats such as GPTQ and AWQ are also supported.

Use cases

• Comparing responses from many LLMs side by side
• Serving a self-hosted chatbot with a web UI
• Evaluating chatbot quality with MT-Bench