BIG-bench

BIG-bench, the Beyond the Imitation Game Benchmark, is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities. It contains more than 200 tasks, summarized by keyword and by task name, aimed at researchers who want to measure and compare language model performance across a wide range of problems. The repository was archived by its owner and is now read-only.

Users install BIG-bench by cloning the repository and installing it, or via pip, then use SeqIO to load and inspect BIG-bench JSON tasks or evaluate a model such as t5x against them. Tasks are either JSON files that define input/target or multiple-choice examples, or programmatic tasks defined in Python that can query a model over several rounds, using each response to inform the next query.

BIG-bench Lite is a canonical subset of 24 diverse JSON tasks, designed to be far cheaper to evaluate than the full set of more than 200 tasks, with a leaderboard of model performance maintained from submitted score files. New tasks can be created from templates such as simple_arithmetic_json or simple_arithmetic, tested with pytest, and evaluated against publicly available models using a provided evaluation script before being submitted as a pull request.

BIG-bench requires Python 3.5 through 3.8 and pytest for running automated tests, and is released under the Apache-2.0 license, free to use. Every task file includes a "canary" string that must not be edited, to prevent benchmark tasks from leaking into web-scraped training data. The repository was archived by its owner on Apr 17, 2026, and no longer accepts new contributions.

Use cases

• Evaluating a t5x model on a BIG-bench JSON task
• Benchmarking model performance cheaply using BIG-bench Lite