A self-hosted RustDesk server is two small containers, five open ports, and about 30 minutes — after which every remote-desktop session in your life routes through hardware you control instead of someone else's relay, and the phrase "commercial use suspected" exits your vocabulary permanently. I've run one on a $5 VPS for two years, supporting three family machines and my own fleet. Here is the entire setup, including the parts the quickstart skips.

Why bother when the public servers work

RustDesk's clients work out of the box through the project's public servers, which is a fine way to evaluate it. Three reasons to graduate. Privacy: the rendezvous server sees your device IDs, IP addresses, and online status, and relayed sessions pass through infrastructure you can't audit — encrypted, but through it. Performance: public relays are shared and can be far away; your own relay on a VPS near you is consistently faster for relayed sessions. Reliability: you're not exposed to public-server load or reachability. The commercial context is even simpler — TeamViewer's per-seat pricing starts around €24/month, and RustDesk's server costs a VPS you probably already have.

The architecture in one paragraph

Three parts. Clients run on every machine. hbbs, the rendezvous server, is the phonebook: clients register their ID with it, and when you connect to a machine, hbbs introduces the two ends and helps them punch through NAT into a direct peer-to-peer session. hbbr, the relay, is the fallback: when NAT traversal fails (symmetric NAT, strict firewalls — in practice maybe 10–20% of session pairs), the whole session streams through it. Direct sessions cost your server almost nothing; relayed ones cost real bandwidth, which is the number to watch below.

The server: one compose file

On any always-on Linux box with a public IP — a $5 VPS is ideal:

services:
  hbbs:
    image: rustdesk/rustdesk-server:latest
    command: hbbs -k _
    network_mode: host
    volumes:
      - ./data:/root
    restart: unless-stopped
  hbbr:
    image: rustdesk/rustdesk-server:latest
    command: hbbr -k _
    network_mode: host
    volumes:
      - ./data:/root
    restart: unless-stopped

Firewall rules: allow 21115/tcp, 21116/tcp+udp, 21117/tcp inbound (add 21118–21119/tcp only if you'll use the browser-based web client). 21116/udp is the rendezvous heartbeat, 21117/tcp is the relay itself. The containers together idle under 50 MB of RAM — the catalogue's 256 MB figure is generous.

On first start, hbbs generates an ed25519 keypair in ./data. The public key — the contents of id_ed25519.pub, a short base64 string — is your server's identity, and you'll paste it into every client.

The -k _ flag is your bouncer

That flag makes both services reject any client that doesn't present your public key. Without it, your server accepts anyone, and open RustDesk servers get found and used — there are scanners that do nothing else, and your relay bandwidth becomes a free public utility. With key enforcement on, a stranger who finds your ports gets refused at the handshake. This is the single most important line in the setup; if you copied a compose file from somewhere that lacks it, add it and restart. It costs you nothing because your clients carry the key anyway.

Configuring clients (and keeping them configured)

On each client: Settings → Network (tap the padlock to allow edits), set ID server to your server's address (DNS name beats raw IP — you can move the VPS later without touching every client), leave relay and API blank, and paste the public key into Key. The home screen's status indicator turns green with "Ready". Do one real test between two machines on different networks — phone on cellular, laptop at home — so you know both the direct and relay paths work before the day a parent is on the phone mid-crisis.

For the family-support use case, set a permanent password on the machines you'll access unattended (Settings → Security), store those passwords in your password manager, and leave click-to-approve enabled on machines that have a human at them. RustDesk also supports pre-configured builds and command-line config flags if you're deploying more than a handful of machines and want the server settings baked in so nobody can accidentally reset to public servers.

The bandwidth math for relayed sessions

A relayed session streams the screen through your VPS in both directions. Rough sustained rates at default quality: a mostly-idle 1080p desktop runs 1–4 Mbps; active scrolling or video pushes 5–15 Mbps. An hour of active relayed support is therefore roughly 2–7 GB of VPS transfer. Against a typical 1–2 TB monthly VPS allowance, a household's worth of support sessions is nothing — but if you plan to work a full day over the relay regularly, either pick a VPS with generous transfer or confirm your sessions go direct (the connection indicator tells you). Direct sessions bypass the relay entirely and cost the server a few kilobytes of introduction traffic.

Where RustDesk fits next to a VPN

If every machine is yours, technical, and always on your Tailscale network, you may not need a remote desktop stack at all — RDP or VNC over the tailnet covers it, as laid out in remote access three ways. RustDesk earns its place where that model breaks: the other end is a person who can read you a 9-digit ID but will never install a VPN; the machines are other people's and don't belong on your tailnet; or you want one tool that behaves identically across Windows, macOS, Linux, iOS, and Android. In my house the split is exactly that — tailnet for my machines, RustDesk for everyone I support.

Bottom line

Two containers with -k _, five firewall rules, clients pointed at a DNS name with the key pasted in, permanent passwords vaulted, one cross-network test. Total cost: half an hour and a corner of a VPS. For that you get TeamViewer-class remote support where the rendezvous data, the relay traffic, and the uptime are all yours — which is the whole self-hosting bargain in miniature.