When do you need human annotation and small-model fine-tuning?

Why prompt and RAG are not always enough

Knowledge Q&A is mostly about finding the right material and composing an answer — RAG is usually the right first bet. But when production needs machine-parseable structure (fixed fields, closed enums, evidence spans copied from source text, PII flags), prompt constraints drift: valid JSON one day, extra fields the next; citations sometimes accurate, sometimes invented.

The gap is often not “does the model know?” but “can it reliably follow the business schema?” Small-model fine-tuning (typically LoRA / QLoRA) trains output shape and rules — a different path from indexing documents in a vector store.

Decision order: do not skip the earlier steps

Prompt / system instructions — lowest cost; zero or few-shot.
RAG — first choice for knowledge Q&A with citations.
Teacher pre-label + human Gold — complex schema, evidence spans, audit trail.
LoRA fine-tuning — holdout Gold exists, training pool is sufficient, baseline shows room to improve.
Larger models / cloud APIs / preference alignment — when LoRA is not enough or the SOW requires it.

Without a frozen schema, a signed acceptance bar from the business owner, and reviewable samples, fine-tuning will not fix everything.

Eight generic steps: corpus to comparison report

FFDE abstracts vertical SLM work into eight repeatable steps (reviews, ticket classification, clause extraction, and similar tasks):

Corpus intake & registry — provenance, de-identification, deduplication; no secrets in Git.
Freeze schema & rubric — fields, enums, evidence rules; business owner sign-off.
Teacher batch pre-label — strong model for speed; log model version and settings.
Automated validation & teacher audit — structural validity, evidence substrings, PII rules; failures to priority queue.
Human Gold review — field-by-field work; reference drafts read-only — no copy-paste.
Gold freeze & holdout lock — block by sample id and content hash; version the holdout file.
Training split + LoRA — teacher labels only, excluding holdout; leakage self-check before train.
Baseline vs fine-tuned eval — same Gold set, same metrics; report in the acceptance evidence pack.

See the methodology page — Vertical SLM for the customer-facing summary.

Three data roles: never mix them

Teacher pre-labels — may enter the training pool (after holdout exclusion); not evaluation ground truth.
Human Gold — holdout evaluation only; never training labels.
Reference drafts — read-only for annotators; do not copy into Gold fields.

Break any rule and you either train on test questions or poison Gold with teacher bias — metrics become meaningless.

Who labels? How do you ensure quality?

High-volume field work is often done by interns, part-timers, or outsourced teams — that is normal in the industry. Schema design, high-risk semantics, and acceptance sign-off stay with domain experts and the business owner.

FFDE recommends at least these mechanisms for a pilot:

Calibration before scale-up — 10–20 items done together; align on ambiguity.
Validate on save — JSON, enums, evidence substrings must pass before submit.
Honeypot items — known-answer rows in the queue; pause annotators below threshold.
Dual annotation & adjudication — double-label a sample; experts resolve conflicts.
Expert spot checks — 5–10% per batch; feed errors back into the rubric.
Batch freeze — ship batch 1 before expanding; avoid thousands of rows under a wrong rubric.

Quality comes from process and gates, not from hoping cheaper labor tries harder.

Holdout and acceptance: when is fine-tuning “real”?

After Gold freeze, order is fixed:

Run baseline on Gold with the base model (no adapter).
Train LoRA on non-holdout teacher labels.
Evaluate again on the same Gold set — compare JSON validity, field accuracy, PII match, and agreed metrics.

Synthetic pilot corpora validate the pipeline only — not production claims. Formal projects use customer de-identified real data with agreed sample size and acceptance thresholds.

Common mistakes

Treating teacher output as Gold — locks evaluation and training to teacher bias.
Random train/test split without content-hash blocking — duplicate text under new ids leaks into training.
Claiming fine-tuning wins without baseline — same eval set, same metrics, side by side.
Expecting general intelligence from tiny data — LoRA improves vertical format and rules, not omniscience.
Handing over only an adapter folder — deliver comparison report, failure cases, data boundaries, and run config.