Prompt Engineering vs Fine-Tuning: Which Strategy to Choose
Start with prompting, use RAG for changing source knowledge, and fine-tune only for a measured behavioural gap. A decision framework with dated evidence and reproducible evaluation.
Use prompt engineering when an existing model can perform the task with clear instructions and examples. Use RAG when answers need current, private or traceable source material. Consider fine-tuning when a stable behavioural gap remains, representative training data is available and a held-out evaluation shows a useful quality, latency or cost improvement. These approaches can be combined.
The decision is about the failure you need to correct. A missing policy document, inconsistent JSON output and expensive inference are different problems, even if each appears as an unsatisfactory model response.
The Decision Framework
Three dimensions still structure the choice:
- Domain stability: how often do source knowledge and business rules change?
- Data readiness: can you produce representative, correctly labelled training and acceptance examples without leakage between them?
- Operating constraints: what are the quality, latency, cost, confidentiality and access requirements?
Set the acceptance criteria before selecting a method. Keep the test set outside prompt and training iterations, and include cases requiring abstention or human review.
When Prompt Engineering Is Sufficient
Prompting is a sensible baseline when the model already has the required capability and the main gap is instruction, context or output structure. Start with a versioned task description, clear examples and an explicit output contract.
There is no universal percentage of fine-tuned quality that prompting reaches, and no fixed example-count threshold separating the two methods. Prompt tokens also have an inference cost. Compare performance on your task rather than treating prompting as a zero-cost option.
An instruction alone does not provide a current company policy, verified access to private documents or a reliable factual reference. Supply authorised sources when the task requires them and evaluate the answer against those sources.
When to Fine-Tune
Fine-tuning is worth testing for repeated, stable behaviour that remains difficult to obtain through the baseline: terminology, routing, a constrained output style or consistent task execution.
Data quality and coverage matter more than a generic promise that hundreds or thousands of examples will be sufficient. Train and evaluate on separate cases, including the difficult categories. Hosting and data handling determine confidentiality; fine-tuning itself does not create that guarantee.
A chat-style training record for a simple routing task can look like this:
training_example = {
"messages": [
{"role": "system", "content": "Classify the request. Return JSON with one label: billing, support, sales, or abstain."},
{"role": "user", "content": "I cannot sign in after resetting my password."},
{"role": "assistant", "content": '{"label":"support"}'},
]
}The record format must match the chosen training stack. The example describes a labelled case; it does not launch training or establish how many cases the task needs.
Dated Results: What the Research Actually Shows
| Study and date | Reported result | Boundary of the evidence |
|---|---|---|
| LoRA, October 2021 revision | Up to 10,000 times fewer trainable parameters and three times lower GPU memory requirements in the stated GPT-3 175B comparison | Parameter-efficient adaptation; not a universal latency or task-quality result |
| InstructGPT, March 2022 | Human evaluators preferred the 1.3B InstructGPT outputs to 175B GPT-3 outputs on the study's prompt distribution | Supervised adaptation plus human-feedback training; not proof that supervised fine-tuning alone always wins |
| QLoRA, May 2023 | Fine-tuning a 65B model on a single 48 GB GPU in the reported setup | Memory feasibility for that method and setup; not a production sizing or training-time promise |
These results motivate controlled experiments. They do not predict the result on your corpus, annotations or hardware.
When to Use RAG Instead of Fine-Tuning
| Need | Fine-tuning | RAG |
|---|---|---|
| Repeated response style or task behaviour | Can adapt the model | Primarily supplies context; instructions still shape behaviour |
| Frequently changing factual sources | Requires a new update process; freshness is not automatic | Sources can be updated through the retrieval pipeline |
| Large proprietary document collection | Training is not a document-access system | Search and context selection are its central role |
| Source traceability | Does not inherently provide citations | Can supply references; answer support still needs checking |
| Access-control changes | Model weights do not enforce document permissions | Retrieval can enforce verified permissions if designed correctly |
The practical distinction is to adapt behaviour with training when justified and provide changing factual knowledge through controlled retrieval. Neither approach guarantees correctness.
A Reproducible Evaluation Example
The synthetic valid fixture produces 3/3 schema-valid outputs and 3/3 correct labels. A fixture containing plain text instead of JSON, a wrong label and an unsupported label produces 1/3 schema-valid outputs and 0/3 correct labels.
Reproduction checked on 12 November 2026. These are evaluator checks, not outputs from a prompted or fine-tuned model. They show that format compliance and task correctness are different measurements.
For an actual comparison, export outputs from your baseline prompt and candidate into one JSON file:
{
"cases": [{"id": "q1", "expected": "support"}],
"runs": {
"prompt-v1": {"q1": "{\"label\":\"support\"}"},
"candidate-v1": {"q1": "{\"label\":\"support\"}"}
}
}Use the same held-out cases for both runs. Measure latency and cost from the real inference calls separately, retain model and prompt revisions, and report sample size and per-category errors. These checks do not measure faithfulness, safety or open-ended answer quality.
Comparative Cost Matrix
| Strategy | Initial work | Inference costs to measure | Maintenance |
|---|---|---|---|
| Prompting | Task specification, examples, evaluation | Model, input/output tokens, retries | Prompt and regression evaluation |
| RAG | Ingestion, search, permissions, evaluation | Retrieval, re-ranking, context and generation | Corpus, index and access changes |
| Fine-tuned hosted model | Data preparation, training, evaluation | Candidate-model usage and any hosting charges | Dataset, model versions and regression checks |
| Fine-tuned self-hosted model | Training plus serving platform | Reserved capacity, utilisation and operations | Model and infrastructure lifecycle |
Only compare cost per completed task at an acceptable quality level. A smaller model can be cheaper, but training, idle capacity, retries and engineering work can change the conclusion.
The Combined Strategy
A production system can combine prompts, retrieval and an adapted model. Introduce the combination in response to measured failures: establish a prompting baseline, add authorised retrieval for source knowledge, then test adaptation for a remaining stable behaviour gap.
The release decision should be supported by an evaluation result and a rollback plan. The system also needs to handle the expected volume of requests: our article on the pitfalls of LLM quota management explains why adding resources does not always increase capacity. Alien6's AI architecture and production engineering offering covers scoping, evaluation and implementation of the chosen approach.