Firecrawl

Last reviewed Aug 26, 2026 · 832 words

The single thing to understand before self-hosting Firecrawl: the open-source repo is the core crawl-and-convert engine, and the cloud product's anti-bot machinery — rotating proxies, stealth browsing, the infrastructure that gets past Cloudflare challenges — largely is not in it. Against ordinary sites (docs, blogs, product pages, your own properties) the self-hosted engine does exactly what the 172,000-star reputation promises: clean Markdown out of messy HTML, whole sites crawled from one API call. Against heavily defended targets, the cloud version will succeed where your instance gets blocked, and no amount of tuning changes that.

What the engine actually does well

Firecrawl's pitch is that it returns LLM-ready output rather than raw HTML. Hit /scrape with a URL and you get back Markdown with navigation, cookie banners, and boilerplate stripped; hit /crawl and it discovers and processes every reachable page under a path, no sitemap required. Because it drives a real headless browser, JavaScript-rendered content — the thing that breaks classic requests-and-parse scrapers — comes through. For RAG ingestion this is the whole value: you want one clean document per page, not 40 KB of div soup, and doing that transformation well across arbitrary sites is harder than it sounds. There is also structured extraction (point it at a page with a JSON schema), but that path calls an LLM, so budget for an API key or a local model behind it.

Plan for 2 GB and a browser that eats it

The catalogue's 2 GB minimum is honest but tight. The self-hosted stack is a Docker Compose arrangement: the TypeScript API service, a Redis queue, and a Playwright-based browser service, with the API listening on port 3002 by default. The browser is where the memory goes — each concurrent page render holds a Chromium context, and a crawl of a few hundred JavaScript-heavy pages at high concurrency will blow past 2 GB. Keep concurrency low (start with 2 or 3 workers) on a small box and the footprint stays civil. There is no database to manage, which makes this one of the easier stateful-looking services to run: lose the containers and you lose nothing but in-flight jobs.

The self-hosted gaps, stated plainly

Three things the cloud tier does that your instance won't, or won't do as well. First, the anti-bot layer described above — self-hosted Firecrawl presents as plain headless Chromium, and sites that fingerprint browsers treat it accordingly. Second, some newer endpoints and features land in the cloud service first, and a few never ship to the repo at all; the project is open about the split, and the self-host docs list what's supported. Third, you get no managed proxy pool, so a big crawl comes from one IP address, which is both a rate-limiting problem and a politeness problem. If your use case is scraping other people's well-defended sites at scale, the honest answer is that the paid product (or Crawl4AI, which is built for local operation from the start) may fit better.

AGPL is a licensing decision, not a footnote

Firecrawl is AGPL-3.0. For a homelab or an internal data pipeline this changes nothing. If you are embedding it in a product that users interact with over a network, the AGPL's source-provision terms apply, and that is precisely why the company sells a commercial cloud offering. Read the license before you build a business on the self-hosted engine; plenty of teams do use it internally without issue, and internal use is the comfortable case.

Crawl like someone who wants to keep crawling

A whole-site crawler with a headless browser can hammer a small server harder than you intend. Respect robots.txt, set the crawl's page limit and depth rather than accepting defaults, and put a delay on anything that isn't your own infrastructure — the arguments in robots.txt and AI crawlers apply doubly when you are the crawler. And if what you actually need is "tell me when this page changes" rather than "give me every page," changedetection.io does that job with a fraction of the resources.

What I'd do

Self-host it if your targets are cooperative: documentation sites, your own products, public content without bot defenses. Docker Compose on a box with 4 GB rather than the minimum 2, concurrency capped, output flowing straight into your embedding pipeline. Test your ten hardest target URLs in the first hour — that tells you immediately whether the open-source engine covers your sources or whether you are in cloud-tier territory. For an internal RAG corpus built from sites you have every right to crawl, the self-hosted engine is the best Markdown-quality-per-watt option I know of; for adversarial scraping at scale, pay someone whose job is the arms race.