Fine-Tuning: The Cases Where It Still Wins

A client came to us last spring with a budget line item that said “fine-tune our own model” and a Notion doc full of vendor quotes. Their actual problem was that GPT-class models kept writing support replies in the wrong tone and occasionally inventing a refund policy. We shipped a retrieval index, a tighter system prompt and a 60 line output validator. Total cost: three days. No training run.

Six weeks later we fine-tuned a model for the same client. Different problem: classifying 40,000 inbound tickets a day into 23 categories, where a frontier model was costing them more per month than a junior hire and adding 900ms to a flow that needed to feel instant.

That’s the shape of it. Fine-tuning an LLM is not dead, and it was never a general-purpose upgrade. It’s a specific tool that wins decisively in a handful of situations and wastes your money in most others. Here’s how we decide, what we actually run, and where we’ve been burned.

What fine-tuning actually changes

Supervised fine-tuning adjusts weights so that, given inputs that look like your prompts, the model’s next-token distribution shifts toward outputs that look like your examples. That’s the whole mechanism. It is very good at moving how a model responds and quite bad at reliably installing what it knows.

Facts learned during fine-tuning are stored diffusely and recalled unreliably, especially when they contradict anything in pretraining. Give a model 300 examples containing your 2026 pricing and it will happily produce 2024 pricing the moment your prompt phrasing drifts from the training distribution. Retrieval puts the fact in the context window where the model can just read it. That’s a different reliability class entirely, and it’s why we build most “AI that knows our stuff” projects as a retrieval system over your own documents rather than a training job.

What fine-tuning does move, hard and cheaply: tone, structure, length discipline, refusal behaviour, label boundaries on a taxonomy only you use, and the model’s default when instructions are ambiguous. These are behavioural properties. They compress into weights beautifully.

When to fine tune: the four cases that still win

1. Classification and extraction at volume

This is the strongest case, and it barely gets written about because it isn’t glamorous. Ticket routing, lead scoring, document field extraction, moderation triage, spam classification with your definition of spam. A fine-tuned 3B or 8B model with a few thousand labelled examples routinely matches or beats a frontier model with an elaborate prompt on these tasks. It runs in 40ms to 80ms instead of 600ms to 1,500ms and costs a fraction per million tokens. On the ticket project, the fine-tuned 8B beat the prompted frontier model on macro F1 by a few points, mostly because it learned the four categories humans confused with each other.

2. Distillation into a model you control

You have a prompt against a frontier model that works. It costs too much, latency is bad, or legal won’t accept the data leaving your VPC. Generate 5,000 to 20,000 input/output pairs with the expensive model, have a human review a sample, then train a small open-weights base on them. You’re not trying to reproduce general intelligence, only this one narrow behaviour, and narrow behaviours distil remarkably well. Check the provider’s terms first: some explicitly forbid training competing models on their outputs, and the rules differ between vendors and between their tiers.

3. House style that resists prompting

If you can describe the style in 200 words of prompt, prompt it. Fine-tuning earns its keep when the style is tacit: an editorial voice with rules nobody has written down, a legal drafting convention, a clinical note format where the ordering is load-bearing. The tell is when your system prompt has grown past 1,500 tokens of “don’t do X” rules, each added after a specific failure, and adding rule 24 breaks rule 9. That prompt is a dataset that hasn’t been converted yet.

4. Tool calling against ugly schemas

Base models are trained on generic function calling. If you have 60 internal tools with overlapping names, or a schema with nested enums and 15 optional fields, selection accuracy degrades badly. Fine-tuning on real traces of correct tool selection fixes routing errors that no amount of prompt engineering resolved for us. Note the ceiling though: this fixes which tool and how to shape arguments, not whether the tool should have been called at all in a weird edge case.

Where it loses, reliably

• Knowledge that changes. Prices, staff, policy, inventory, anything with a date. Retrieval, every time.
• Fewer than roughly 200 clean examples. Below that you’re mostly teaching format, which few-shot prompting does for free and without a deploy pipeline.
• Valid JSON output. vLLM, llama.cpp, and every major hosted API now support grammar or JSON schema constrained decoding. Structural validity is a decoding problem, not a weights problem. Fine-tuning for it is 2023 thinking.
• Improving reasoning. Training an 8B on 3,000 examples of your hardest multi-step problems will not make it reason like a frontier model. It will make it produce confident-looking output in the right shape. That failure mode is worse than the original, because it’s harder to spot.
• Tasks you can’t evaluate. If you have no measurable definition of a better answer, a training run just moves the model somewhere you can’t characterise.

The dataset is the entire project

Every fine-tune we’ve regretted failed at the data layer, not the training layer. The training is a config file. The data is the work.

Practical rules that have survived contact with real projects:

• Hold out the eval set first. Split before you clean, before you augment, and never let a near-duplicate cross the boundary. Fuzzy-dedup with MinHash or simple normalised-string hashing; leaked duplicates are the number one source of eval numbers that don’t survive production.
• One output style per task. If half your examples answer in two sentences and half in six paragraphs, you’re training variance. Pick one and rewrite the rest.
• Include the hard negatives. For classification, deliberately over-sample the pairs humans confuse. For refusals, include real out-of-scope questions with the exact refusal you want.
• Mask the prompt. Train on completion tokens only unless you have a specific reason not to. Most frameworks default to this now; verify rather than assume.

The format is boring and that’s the point. One JSON object per line:

{"messages":[{"role":"system","content":"Classify the ticket. Reply with one label only."},{"role":"user","content":"Card declined twice but my bank says it went through"},{"role":"assistant","content":"billingduplicatecharge"}]}
{"messages":[{"role":"system","content":"Classify the ticket. Reply with one label only."},{"role":"user","content":"How do I export my team's data before we cancel?"},{"role":"assistant","content":"account_offboarding"}]}

Keep the system prompt in training identical to the one you’ll use in production. Byte for byte. Drift between the two is the silent killer: the model was tuned on a distribution you’re no longer sending it.

Actually running it

Use LoRA or QLoRA. Full fine-tuning of a 7B to 14B model is justified when you’re changing the model broadly (a new language, a genuinely different domain vocabulary) and you have tens of thousands of examples. For everything described above, a rank 16 adapter on an 8B base gets you within noise of full fine-tuning at a fraction of the VRAM, and you can hot-swap adapters at serve time.

An Axolotl config we’ve used close to verbatim on several projects:

cat > qlora-8b.yml <<'YAML'
base_model: meta-llama/Llama-3.1-8B-Instruct
loadin4bit: true
adapter: qlora

datasets:

path: ./data/train.jsonl
    type: chat_template
    field_messages: messages
valsetsize: 0.0          # we hold out our own eval file, never a random split
output_dir: ./out/ticket-router-v3

sequence_len: 2048
sample_packing: true        # big throughput win; disable if examples are near max length
padtosequence_len: true

lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
loratargetmodules: [qproj, kproj, vproj, oproj, gateproj, upproj, down_proj]

microbatchsize: 4
gradientaccumulationsteps: 4
num_epochs: 3
optimizer: adamwbnb8bit
lr_scheduler: cosine
learning_rate: 0.0002
warmup_ratio: 0.03
bf16: auto
gradient_checkpointing: true
save_strategy: epoch        # keep every epoch, epoch 2 often beats epoch 3
YAML

accelerate launch -m axolotl.cli.train qlora-8b.yml

A few things the docs underplay. Three epochs is a starting point, not a target: on datasets under 1,000 examples we usually find epoch 2 is the best checkpoint and epoch 4 has started memorising. Learning rate 2e-4 suits rank 16; if you push rank to 64 or 128, drop it toward 1e-4. Keep lora_alpha at roughly twice the rank unless you’re deliberately tuning effective scale. Save every epoch, because the loss curve will not tell you which checkpoint is best on your actual task.

If you’d rather not touch GPUs, hosted fine-tuning is a legitimate choice for case 1 and case 3, and it’s how we start most classification pilots:

FILE_ID=$(curl -s https://api.openai.com/v1/files \
  -H "Authorization: Bearer $OPENAIAPIKEY" \
  -F purpose="fine-tune" -F file="@train.jsonl" | jq -r .id)

curl https://api.openai.com/v1/fine_tuning/jobs \
  -H "Authorization: Bearer $OPENAIAPIKEY" \
  -H "Content-Type: application/json" \
  -d "{\"trainingfile\":\"$FILEID\",\"model\":\"gpt-4.1-mini-2025-04-14\",
       \"hyperparameters\":{\"n_epochs\":3}}"

The trade-off is stark: zero infrastructure, but you’re renting a custom model on someone else’s deprecation schedule, and you cannot take the weights with you.

Evaluation, or you’re just guessing

Run your eval set against the base model before you train anything. That baseline number is the only thing that makes the post-training number meaningful. Skipping it is the most common mistake we see in client projects that arrive half-finished.

Pick the metric that matches the task. Classification: per-class precision and recall plus macro F1, because overall accuracy hides the rare class you actually care about. Extraction: exact match per field, then a normalised match for dates and numbers. Generation: pairwise preference against the baseline, judged blind, with the two candidates shuffled per item. LLM judges have real position bias, so if you don’t randomise order you’ll measure the layout of your prompt, not the quality of your outputs.

Keep 30 to 50 examples as a regression suite that never enters training and gets run on every candidate checkpoint. Ours includes the specific failures that caused the project to exist, plus a handful of adversarial inputs. It’s the same discipline we apply to AI-generated code we have to maintain: the output is only as trustworthy as the check you run on it.

Watch for two specific regressions. Instruction-following on tasks outside your training distribution degrades quietly after aggressive fine-tuning. So does the model’s willingness to say “I don’t know”, because almost no one includes uncertainty examples in their dataset, so the model learns that every input has a confident answer.

The economics and the maintenance bill

The training cost is trivial. An 8B QLoRA run over a few thousand examples finishes in tens of minutes on a single 80GB card, which is single-digit dollars on most rental providers. The real costs are the labelled dataset (weeks of somebody’s attention), the eval harness, and serving.

Serving is where the maths flips. A frontier model on a per-token API has zero fixed cost, so at low volume it always wins. A self-hosted 8B carries a fixed GPU bill whether you send it 100 requests or 10 million. Break-even in the projects we’ve run tends to arrive somewhere in the high hundreds of thousands to low millions of requests per month. It arrives much sooner if latency has business value or if data residency rules out the API entirely. Run the numbers with your actual token counts before committing. Multi-LoRA serving in vLLM helps considerably: one base model can host a dozen adapters and amortise that fixed cost across several tasks.

Then budget for the treadmill. Base models improve every few months, and a fine-tune that beat the frontier model in March may lose to next quarter’s default. That’s fine if retraining is a command you run against a versioned dataset. It’s a crisis if the dataset lives in one person’s laptop. Version the data, the config and the eval set together, tag the adapter with the commit, and you can re-run the whole thing against a new base in an afternoon.