crawl4ai

Crawl4ai is an open-source web crawler and scraper designed for large language models and AI agents. It converts pages into clean Markdown suitable for AI workloads and data pipelines. The project emphasizes running the tool yourself in an open-source fashion, or using Crawl4AI Cloud with a single key to access cloud features. The repository presents both self-hosted and hosted options as part of its offering.

Two main usage models are described: running the tool yourself with an installation workflow, or using the cloud service that handles the browserless crawling tasks. The cloud option includes a single key to access its features and a set of endpoints for scraping, extracting, searching the web, and answering questions with sources. The self-hosted path provides installation commands and Docker/server options.

The tool provides Markdown generation to represent scraped content, supports structured data extraction, and offers browser control through persistent profiles, proxies, and various rendering engines. It also includes caching, multi-URL processing, and supports self-hosting via Docker or cloud access for different deployment needs.

Plans include a cloud option and self-hosted options, with mentions of free credits and paid usage. The cloud path advertises a credit-based pay-as-you-go model, while self-hosting is available via Docker with API tokens and local server setup.

Use cases

• Convert any page to Markdown for LLMs