gallery-dl

Last reviewed Aug 26, 2026 · 731 words

gallery-dl will not show you a single picture. It is a Python command-line downloader that mirrors image galleries, user profiles and collections from a few hundred sites into folders on your disk, with the metadata preserved as JSON sidecars, and then gets out of the way. The self-hosting move is to run it on a schedule against the accounts and albums you care about, point a real gallery such as Immich at the output, and never think about a site shutting down again. The one setting that separates a tidy archive from 40 GB of duplicates is --download-archive, so it comes first.

The archive file is the whole trick

Every download is recorded in a small SQLite file when you pass --download-archive. On the next run gallery-dl skips anything already listed, which turns a one-off scrape into an incremental sync:

pip install gallery-dl
gallery-dl --download-archive /srv/archive/gdl.sqlite3 \
  --write-metadata \
  -d /srv/archive \
  https://example-site/user/somephotographer

Keep that SQLite file with the downloads and back it up alongside them. Lose it and the next run re-fetches everything, which is slow, wasteful and, on sites with rate limits, the quickest way to get your account throttled. --write-metadata writes a .json next to each file with the title, tags, upload date and source URL, which is the difference between an archive and a pile.

Put the options in a config file, not in cron

The command-line flags are fine for a test. For a scheduled job, the same settings belong in ~/.config/gallery-dl/config.json, where you can also set per-site behaviour:

{
  "extractor": {
    "base-directory": "/srv/archive",
    "archive": "/srv/archive/gdl.sqlite3",
    "postprocessors": [{"name": "metadata", "mode": "json"}],
    "sleep-request": 3.0
  }
}

Then the cron line is just gallery-dl -i /srv/archive/urls.txt, reading one URL per line from a file you edit when you find something new. sleep-request (a pause in seconds between requests) is the polite setting; a 2 to 5 second gap keeps most sites happy, and most bans I have seen came from someone running with none. At 128 MB of RAM gallery-dl is happy on the smallest box you own, so run it on the same machine that holds the archive rather than somewhere that then has to sync across.

Logged-in sites need cookies, and cookies expire

Anything behind a login wants either credentials in the config or a cookies file. --cookies-from-browser firefox reads them straight from a local browser profile, which is convenient on a desktop and useless on a headless server. On the server, export a cookies.txt once and reference it with "cookies": "/srv/archive/cookies.txt". Expect to refresh it every few months; the symptom of an expired session is a run that "succeeds" with zero new files. Watch the exit code and log output from cron rather than assuming silence is good news.

What it is not, and what to put next to it

It is not a gallery, a deduplicator or a video downloader. For video, yt-dlp is the sibling project with the same conventions, and gallery-dl can hand video URLs through to it via its ytdl extractor. For viewing, Immich's external-library feature can index /srv/archive read-only, so the mirrored photos appear in the app with their dates intact and nothing is moved. PhotoPrism does the same with its originals folder, and the photos category covers the trade-offs between them. Keep gallery-dl's folder separate from your own camera roll; the sidecars and site-specific naming are useful for provenance but ugly in a family album.

What I'd do

A dedicated /srv/archive on the box that runs Immich, gallery-dl installed with pip, the config above with a 3-second sleep-request, and a nightly cron job reading from urls.txt with output logged to a file that gets grepped for errors. The SQLite archive and the JSON sidecars go into the same restic backup as everything else. Add an Immich external library pointing at the folder, read-only. Total effort is an hour to set up and a few minutes a quarter to refresh cookies, in exchange for an archive that survives any site deciding to close.