Skip to main content
← ArticlesLire en français
·Alien6 Research

Prompt Engineering vs Fine-Tuning: Which Strategy to Choose

Start with prompting, use RAG for changing source knowledge, and fine-tune only for a measured behavioural gap. A decision framework with dated evidence and reproducible evaluation.

LLMPrompt EngineeringFine-Tuning

Use prompt engineering when an existing model can perform the task with clear instructions and examples. Use RAG when answers need current, private or traceable source material. Consider fine-tuning when a stable behavioural gap remains, representative training data is available and a held-out evaluation shows a useful quality, latency or cost improvement. These approaches can be combined.

The decision is about the failure you need to correct. A missing policy document, inconsistent JSON output and expensive inference are different problems, even if each appears as an unsatisfactory model response.

The Decision Framework

Three dimensions still structure the choice:

  1. Domain stability: how often do source knowledge and business rules change?
  2. Data readiness: can you produce representative, correctly labelled training and acceptance examples without leakage between them?
  3. Operating constraints: what are the quality, latency, cost, confidentiality and access requirements?

Set the acceptance criteria before selecting a method. Keep the test set outside prompt and training iterations, and include cases requiring abstention or human review.

When Prompt Engineering Is Sufficient

Prompting is a sensible baseline when the model already has the required capability and the main gap is instruction, context or output structure. Start with a versioned task description, clear examples and an explicit output contract.

There is no universal percentage of fine-tuned quality that prompting reaches, and no fixed example-count threshold separating the two methods. Prompt tokens also have an inference cost. Compare performance on your task rather than treating prompting as a zero-cost option.

An instruction alone does not provide a current company policy, verified access to private documents or a reliable factual reference. Supply authorised sources when the task requires them and evaluate the answer against those sources.

When to Fine-Tune

Fine-tuning is worth testing for repeated, stable behaviour that remains difficult to obtain through the baseline: terminology, routing, a constrained output style or consistent task execution.

Data quality and coverage matter more than a generic promise that hundreds or thousands of examples will be sufficient. Train and evaluate on separate cases, including the difficult categories. Hosting and data handling determine confidentiality; fine-tuning itself does not create that guarantee.

A chat-style training record for a simple routing task can look like this:

training_example = {
    "messages": [
        {"role": "system", "content": "Classify the request. Return JSON with one label: billing, support, sales, or abstain."},
        {"role": "user", "content": "I cannot sign in after resetting my password."},
        {"role": "assistant", "content": '{"label":"support"}'},
    ]
}

The record format must match the chosen training stack. The example describes a labelled case; it does not launch training or establish how many cases the task needs.

Dated Results: What the Research Actually Shows

Study and dateReported resultBoundary of the evidence
LoRA, October 2021 revisionUp to 10,000 times fewer trainable parameters and three times lower GPU memory requirements in the stated GPT-3 175B comparisonParameter-efficient adaptation; not a universal latency or task-quality result
InstructGPT, March 2022Human evaluators preferred the 1.3B InstructGPT outputs to 175B GPT-3 outputs on the study's prompt distributionSupervised adaptation plus human-feedback training; not proof that supervised fine-tuning alone always wins
QLoRA, May 2023Fine-tuning a 65B model on a single 48 GB GPU in the reported setupMemory feasibility for that method and setup; not a production sizing or training-time promise

These results motivate controlled experiments. They do not predict the result on your corpus, annotations or hardware.

When to Use RAG Instead of Fine-Tuning

NeedFine-tuningRAG
Repeated response style or task behaviourCan adapt the modelPrimarily supplies context; instructions still shape behaviour
Frequently changing factual sourcesRequires a new update process; freshness is not automaticSources can be updated through the retrieval pipeline
Large proprietary document collectionTraining is not a document-access systemSearch and context selection are its central role
Source traceabilityDoes not inherently provide citationsCan supply references; answer support still needs checking
Access-control changesModel weights do not enforce document permissionsRetrieval can enforce verified permissions if designed correctly

The practical distinction is to adapt behaviour with training when justified and provide changing factual knowledge through controlled retrieval. Neither approach guarantees correctness.

A Reproducible Evaluation Example

The synthetic valid fixture produces 3/3 schema-valid outputs and 3/3 correct labels. A fixture containing plain text instead of JSON, a wrong label and an unsupported label produces 1/3 schema-valid outputs and 0/3 correct labels.

Reproduction checked on 12 November 2026. These are evaluator checks, not outputs from a prompted or fine-tuned model. They show that format compliance and task correctness are different measurements.

For an actual comparison, export outputs from your baseline prompt and candidate into one JSON file:

{
  "cases": [{"id": "q1", "expected": "support"}],
  "runs": {
    "prompt-v1": {"q1": "{\"label\":\"support\"}"},
    "candidate-v1": {"q1": "{\"label\":\"support\"}"}
  }
}

Use the same held-out cases for both runs. Measure latency and cost from the real inference calls separately, retain model and prompt revisions, and report sample size and per-category errors. These checks do not measure faithfulness, safety or open-ended answer quality.

Comparative Cost Matrix

StrategyInitial workInference costs to measureMaintenance
PromptingTask specification, examples, evaluationModel, input/output tokens, retriesPrompt and regression evaluation
RAGIngestion, search, permissions, evaluationRetrieval, re-ranking, context and generationCorpus, index and access changes
Fine-tuned hosted modelData preparation, training, evaluationCandidate-model usage and any hosting chargesDataset, model versions and regression checks
Fine-tuned self-hosted modelTraining plus serving platformReserved capacity, utilisation and operationsModel and infrastructure lifecycle

Only compare cost per completed task at an acceptable quality level. A smaller model can be cheaper, but training, idle capacity, retries and engineering work can change the conclusion.

The Combined Strategy

A production system can combine prompts, retrieval and an adapted model. Introduce the combination in response to measured failures: establish a prompting baseline, add authorised retrieval for source knowledge, then test adaptation for a remaining stable behaviour gap.

The release decision should be supported by an evaluation result and a rollback plan. The system also needs to handle the expected volume of requests: our article on the pitfalls of LLM quota management explains why adding resources does not always increase capacity. Alien6's AI architecture and production engineering offering covers scoping, evaluation and implementation of the chosen approach.