Anubis

Last reviewed Aug 26, 2026 · 881 words

Put Anubis in front of your Git forge, your wiki, and any other app where every page is a database query, and nowhere else. It is a 64 MB Go proxy that makes each new visitor's browser solve a small SHA-256 proof-of-work puzzle before passing them through, which a laptop finishes in about a second and a crawler fleet fetching 10 million pages cannot afford. It exists because AI crawlers stopped honouring robots.txt, and it went from one person's Gitea to GNOME's GitLab and kernel.org's infrastructure in about a year. The price is that anything without a JavaScript engine (feed readers, API clients, git clone over HTTPS) hits the wall too unless you exempt it, so most of this guide is about exemptions.

The problem: crawlers that fetch every commit diff, twice

A Git forge is the worst possible site to be scraped. Every commit, every diff, every blame view, every file at every revision is a unique URL that costs a database query and a git process to render. A crawler that ignores robots.txt (the robots.txt for AI crawlers post explains why that stopped working) can put a 4-core Forgejo box on its knees from a single residential IP range, and the IPs rotate, so banning them is whack-a-mole. Anubis flips the cost: the crawler has to spend CPU per session, and it has to run a browser to do it, which most don't.

It sits between your reverse proxy and the app

Anubis is not a reverse proxy in its own right; it's a filter your proxy sends traffic through:

services:
  anubis:
    image: ghcr.io/techarohq/anubis:latest
    environment:
      BIND: ":8080"
      TARGET: "http://forgejo:3000"
      DIFFICULTY: "4"
      METRICS_BIND: ":9090"
      SERVE_ROBOTS_TXT: "true"
      POLICY_FNAME: "/data/botPolicies.yaml"
      ED25519_PRIVATE_KEY_HEX: ${ANUBIS_KEY}
    volumes:
      - ./botPolicies.yaml:/data/botPolicies.yaml:ro
    restart: unless-stopped

Your Caddy site block changes from reverse_proxy forgejo:3000 to reverse_proxy anubis:8080, and that's the whole integration. Generate the key with openssl rand -hex 32 and keep it stable; without it Anubis makes a new one each start and every visitor re-solves the challenge after a restart. METRICS_BIND exposes a Prometheus endpoint that tells you how many challenges were issued versus passed, which is how you'll know it's working. Do not publish that port.

Exempt feeds, APIs, and Git clients or they break

The default policy waves through the major search engines by user agent, allows /.well-known/, robots.txt, and favicons, and challenges anything that identifies as a browser. It knows nothing about your app. A first policy file for a forge:

bots:
  - name: git-clients
    user_agent_regex: "^git/"
    action: ALLOW
  - name: forge-api
    path_regex: "^/api/"
    action: ALLOW
  - name: feeds
    path_regex: "\\.(rss|atom)$"
    action: ALLOW
  - name: well-known
    path_regex: "^/\\.well-known/"
    action: ALLOW
  - name: browsers
    user_agent_regex: "Mozilla"
    action: CHALLENGE

Rules match top to bottom. Without the first one, git clone https://... fails with an HTML page where a pack file should be (SSH clones are unaffected, they never touch HTTP). Without the API rule, Renovate, CI runners, and mobile apps break. Feed readers, link-preview unfurlers in Slack and Discord, and monitoring probes are the other regulars; add them as you find them in the metrics. This is the catalogue's "can block benign crawlers" con made concrete: the default is strict, and the first week is spent widening it.

Difficulty 4 costs a human a second and a fleet a fortune

DIFFICULTY is the number of leading zero hex digits the browser must find, so 4 means about 65,000 hash attempts on average. A modern laptop does that in well under a second, a mid-range phone in 1 to 3 seconds, and the result is cached in a cookie for a week, so a real visitor pays once. A scraper paying it per session, across millions of sessions, is spending real money. Resist raising it: 5 is 16 times the work and the phone experience becomes a 30-second loading screen. If 4 isn't enough, the answer is a stricter policy, not a bigger number.

CrowdSec does the other half

Anubis taxes unknown visitors; CrowdSec bans known-bad ones from a shared reputation list before the request arrives. They compose well: CrowdSec drops the IPs everyone has already caught, Anubis handles the residential proxies nobody has. Neither inspects the request body for attacks, which is a WAF's job and a different guide. If you're only going to run one on a forge, run Anubis; the scraper load is the problem you'll actually have.

What I'd do

Anubis between Caddy and every Git forge and wiki I expose, difficulty 4, a stable key, and the policy file above extended over the first week by reading the metrics. Not in front of a blog or docs site that should be indexed freely and costs nothing to serve; the challenge page is friction, and static HTML doesn't need protecting. If one 64 MB container had to stand between my Forgejo and the internet, this is the one, and the CPU graph on the forge the day after installing it is the argument.