Crawl4AI

Last reviewed Aug 26, 2026 · 781 words

Feed an LLM raw HTML and you pay token prices for nav bars, cookie banners, and 40 KB of inline SVG. Crawl4AI exists to stop that: it drives a real browser at a page and hands back clean Markdown with the boilerplate stripped, which is the format every RAG pipeline and agent framework actually wants. It is Apache-2.0, free at any volume, and the obvious self-hosted answer to Firecrawl's per-page credit pricing — with the trade that you operate the browsers yourself.

The output is the product

A basic crawl is four lines of Python:

from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://example.com/docs/page")
    print(result.markdown)

What comes back is not a naive HTML-to-Markdown dump. Because a real browser rendered the page, JavaScript-built content is present, and the content filtering works to separate the article from the chrome around it. On documentation sites, blogs, and news pages the result is usually clean enough to embed directly; the difference in embedding quality between this and raw-HTML scraping is visible in retrieval results, not just token bills. If the destination is a vector store, choosing a vector database covers the other half of that pipeline.

Library or API server: pick by who's calling

Crawl4AI is Python-first (a pip install crawl4ai, then the crawl4ai-setup command to pull down Playwright's browser binaries), and if your pipeline is Python, staying in-process is simplest. The second mode matters for everyone else: the project ships a Docker image that exposes crawling as a REST API, so an n8n flow, a Node service, or any HTTP client can request "this URL as Markdown" from a crawler box you run once and share. In a homelab, the API-server mode is the one I would reach for — one container, one endpoint, every project on the network gets a scraper.

The catalogue's Medium difficulty is about this operational layer, not the code. The library is easy; keeping a fleet of headless Chromium processes healthy under load is where the experience points get spent.

The 2 GB minimum is a browser, not the crawler

Crawl4AI itself is a modest async Python program. The catalogue's 2048 MB floor is Playwright's Chromium instances, and it is a floor: each concurrent browser context costs real memory, and a page heavy with JavaScript costs more. Two sizing rules save you the discovery process. First, throttle concurrency to what your RAM actually supports rather than what asyncio permits — a dozen parallel contexts on a 2 GB box will OOM. Second, give the Docker deployment a hard memory limit so a runaway page kills one crawl, not the host. Sites that fight automation (Cloudflare challenges, aggressive bot detection) are their own arms race; Crawl4AI's real-browser approach clears the easy cases, but do not promise your project a 100% fetch rate on hostile targets.

Use CSS extraction before you pay for LLM extraction

Beyond Markdown, Crawl4AI does structured extraction two ways: schema-based extraction using CSS selectors, and LLM-based extraction where a model pulls fields from the page. The selector route is free, fast, and deterministic — right for any site whose layout you can inspect once. Save the LLM route for genuinely irregular pages, and when you use it, point it at a local model via Ollama before paying a hosted API to read web pages at scale; extraction is exactly the kind of high-volume, low-difficulty task where local models earn their keep in 2026.

Crawl like someone who wants the site to survive

A polite-crawling note, because a self-hosted crawler has no rate-limiting vendor between you and your targets: respect robots.txt, set a real user agent, cap request rates per domain, and cache aggressively so re-runs hit your store instead of their servers. This keeps you legal, keeps your IP unblocked, and is simply the right way to treat sites run by people like the ones reading this directory.

What I'd do

Python project, moderate volume: install the library, run crawl4ai-setup, keep concurrency in the low single digits per 2 GB of RAM, and cache every fetch. Anything multi-project or non-Python: deploy the Docker API server on a box with 4 GB, memory-limited, and treat it as shared infrastructure. Use CSS-schema extraction by default, local-LLM extraction for the messy 10%, and revisit Firecrawl only if operating browsers ever costs you more time than a credit bill would — for most self-hosters it never does.