LoRA Fine-Tuning on a Consumer GPU

A 24GB consumer GPU — an RTX 3090 off eBay for ~$700 or a 4090 — fine-tunes an 8B model with QLoRA in one to three hours on 1,000–5,000 examples, and a 14B overnight. The outcome is decided almost entirely by dataset quality; hyperparameters mostly decide whether you find that out cheaply. And LoRA is the right tool for teaching style, format, and narrow-task behaviour — if you're trying to inject knowledge, stop and read RAG vs fine-tuning first, because you probably want retrieval.

What fits on 24GB

QLoRA loads the base model in 4-bit and trains low-rank adapters in bf16, which collapses memory to roughly weights ÷ 3.5 plus optimizer and activation overhead. In practice, at sequence length 2,048:

Sequence length is the hidden multiplier: activations scale with it, so doubling sequence_len can cost more than moving up a model size. Set it to your dataset's real p95, not the model's maximum. The general VRAM arithmetic applies here too, with training activations replacing inference KV cache as the swing factor.

Dataset prep is the actual work

Five hundred good examples beat five thousand scraped ones — past a few thousand rows for a narrow task, quality and diversity dominate volume. The rules that save reruns: format every example with the model's own chat template (a mismatched template trains the model to emit broken special tokens); deduplicate near-duplicates, or the model memorises your most common row; hold out 10% before looking at anything; and decontaminate — check that no training row is a near-duplicate of your eval set, with exact match plus embedding similarity above ~0.95 as the filter. If you're generating training data with a bigger model, the usual synthetic-data pitfalls all apply: diversity collapses at low sampling temperature, and a teacher's systematic errors become your labels.

A config that works, and the knobs that matter

This Axolotl config is a sane starting point for an 8B on 24GB:

base_model: meta-llama/Llama-3.1-8B-Instruct
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
sequence_len: 2048
micro_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 2e-4
lr_scheduler: cosine
warmup_ratio: 0.03
num_epochs: 3
bf16: true

Ranked by how much they actually matter: learning rate first — 2e-4 is the QLoRA sweet spot, drop to 1e-4 if training loss spikes or outputs degrade into repetition. Epochs second — 2–3 for most sets; small datasets overfit fast, and the symptom is eval loss rising while train loss falls. Target modules third — lora_target_linear: true (all linear layers) reliably beats attention-only targeting; it's the cheapest quality win in the config. Rank barely matters between 8 and 32 for narrow tasks; going past 64 spends VRAM for nothing measurable. For scale: 3 epochs over 3,000 examples at this config takes roughly 2 hours on a 3090 and two-thirds of that on a 4090, so a full change-one-variable experiment cycle fits in an evening. Unsloth is worth knowing as the runtime — roughly 2× faster and meaningfully leaner on VRAM for single-GPU runs (docs), same knobs.

Evaluate before, after, and sideways

Run your task eval on the base model before training — that's the number that tells you whether fine-tuning was worth it, and skipping it is the most common way people fool themselves. After training, run the same harness on the adapter and require a gap that survives 50+ examples, not vibes on five. Then run a sideways check: 20–30 general prompts (reasoning, instruction-following, refusals) compared against the base, because aggressive fine-tunes degrade general behaviour and you want to catch a model that aces your task but has forgotten how to say "I don't know". The harness mechanics — golden sets, judge calibration — are covered in evals before vibes.

Typical honest outcome on a narrow task (classification with house labels, structured extraction, a house writing style): base 8B at 60–75% on your metric, tuned at 85–95%, at which point it replaces a frontier-model call that was scoring similarly for 20× the price.

Shipping the adapter

Two good paths. Merge the adapter into the base weights and quantise to GGUF (llama.cpp's convert_lora_to_gguf.py, or merge in PEFT then convert) for serving via Ollama — one artifact, boring to operate. Or keep adapters separate and serve with vLLM's --enable-lora, which hot-loads multiple adapters over one base model — the right shape when you have several task-specific tunes sharing a GPU. Quantising the merged model costs another 1–3% on your metric; re-run the eval on the exact artifact you deploy, per the usual quantisation caveats.

What I'd do

Spend day one building 1,000 clean examples and an eval set, not training. Train the 8B QLoRA with the config above (2–3 hours on a 3090), compare base vs tuned vs your current production model on the same harness, and make the call on numbers. If the tuned 8B wins, merge, quantise, re-eval the artifact, and ship it behind a flag. Total cash cost if you rent instead of own: about $5–15 of GPU time per full experiment cycle — cheap enough that the only expensive mistake is skipping the baseline eval.