Applied AI 7 min read15 January 2025

Fine-Tuning vs. RAG vs. Prompting: A Decision Framework

Joshua Lamerton

Fine-Tuning RAG LLM Infrastructure AI Architecture Production ML

One of the most common questions in applied AI right now: should I fine-tune, use RAG, or just engineer a better prompt? The answer depends on what problem you're actually solving.

The Prompting Baseline

Always start here. Prompting is zero-cost to experiment, zero-infrastructure to deploy, and increasingly powerful as context windows expand. If prompting gets you 80% of the way there, ask hard questions before investing in anything more complex.

Prompting fails when: you need consistent structured output at scale, your task requires knowledge not in the model's training data, or you're hitting context window limits with your few-shot examples.

When to Use RAG

RAG is the right choice when the primary problem is knowledge currency or specificity — you need the model to reason over documents it wasn't trained on. Customer support over a product knowledge base. Legal research over a specific regulatory corpus. Property questions over real-time listing data.

RAG does not make the model smarter. It gives the model access to better information. Don't confuse the two.

When to Fine-Tune

Fine-tuning is appropriate when you need to change the model's behaviour, not just its knowledge. Style, format, tone, task-specific reasoning patterns. Fine-tuning is expensive, slow to iterate, and requires careful evaluation infrastructure — don't default to it because it sounds impressive.

The Combined Play

The most powerful production systems usually combine all three: a fine-tuned model (for behaviour), a RAG pipeline (for knowledge), and carefully engineered system prompts (for task framing and output constraints). Each layer doing what it's best at.

Clap