29 Jul 2023 AI Engineering 4 min read

RAG vs Fine-Tuning for Company Knowledge

Retrieval-augmented generation or fine-tuning: which puts company knowledge into a language model? A mid-2023 engineering view. We explain what fine-tuning actually changes, how embeddings and vector search work, why retrieval should be the default for facts, and where fine-tuning earns its cost — grounded in Lewis et al. (2020) and current tooling.

The Problem With Parametric Knowledge

A large language model stores what it saw during training in its weights. This parametric knowledge is frozen at the training cutoff, it cannot cite a source, and where a fact is missing the model produces plausible text instead of admitting the gap. Company knowledge — product specifications, contracts, support tickets, internal wikis — is precisely what the model has never seen. Getting it into the system is the central engineering question of applied LLM work in 2023.

Since ChatGPT launched in November 2022, the same question appears in almost every client conversation we have at Blue IT Systems: can we train the model on our data? The honest answer: you can, but you should rarely start there. Two mechanisms compete for this job. They solve different problems, and confusing them is the most common failure we see in project proposals.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

What Fine-Tuning Actually Changes

Fine-tuning continues training a pre-trained model on your own examples. It shifts the weights so the model imitates the distribution of the training data: tone, format, task structure. Parameter-efficient methods have made this affordable. LoRA (Hu et al., June 2021) freezes the base model and trains small low-rank adapter matrices — at GPT-3 scale roughly 10,000 times fewer trainable parameters than full fine-tuning. QLoRA (Dettmers et al., May 2023) fine-tunes a 65-billion-parameter model on a single 48 GB GPU.

What fine-tuning does not do reliably is inject facts. A model fine-tuned on your product manual does not become a lookup table for that manual. It becomes a model that writes like the manual. Recall of specific values stays probabilistic and uncited, and it is frozen at the moment of training: every document revision implies another training run, another evaluation, another deployment.

Retrieval-Augmented Generation Defined

Retrieval-augmented generation takes the opposite route: leave the weights alone and put the facts into the prompt. The term comes from Lewis et al. (2020) at Facebook AI Research. Their architecture coupled a dense retriever (DPR, Karpukhin et al. 2020) with a seq2seq generator (BART) over a vector index of Wikipedia. The generated answers were measurably more specific and more factual than those of the parametric-only baseline. One result matters most in practice: the knowledge index could be replaced without retraining the model.

The production pattern of 2023 is simpler than the original end-to-end model. Split documents into chunks of a few hundred tokens. Embed each chunk and store the vector. At query time, embed the question, retrieve the most similar chunks, and hand them to an instruction-tuned LLM with the explicit instruction to answer only from the supplied context. That instruction reduces hallucination; it does not eliminate it.

Embeddings and Vector Search Basics

An embedding model maps text to a vector such that semantic similarity becomes geometric proximity, usually measured as cosine similarity. OpenAI's text-embedding-ada-002 (December 2022) returns 1,536-dimensional vectors and has cost 0.0001 US dollars per 1,000 tokens since June 2023 — embedding a corpus of ten million tokens costs about one dollar. Open-source models from the sentence-transformers family run on your own hardware, which matters for clients with data-residency requirements.

At scale, exact comparison against every stored vector gives way to approximate nearest-neighbor indexes such as HNSW. Faiss provides the algorithms as a library, pgvector adds vector search to PostgreSQL, and dedicated stores such as Weaviate, Qdrant and Milvus package indexing with filtering. Two honest caveats: chunking strategy dominates retrieval quality, and a retriever that returns the wrong passages fails silently — the generator will answer anyway.

Why Retrieval Wins for Facts

For factual questions the comparison is one-sided. An index update is a database write; a weight update is a training run. Retrieved chunks give you citations you can display next to the answer; weights cannot. Retrieval can enforce per-user permissions at query time; a fine-tuned model reproduces whatever it memorized for anyone who asks. None of this makes RAG accurate by itself — it makes errors visible, attributable and correctable, which is what production systems need.

Criterion	Retrieval (RAG)	Fine-tuning
Updating a fact	Re-index one document	New training run
Source attribution	Retrieved chunks as citations	None
Per-user access control	Filter at query time	Not enforceable
Behavior on missing facts	Can refuse based on context	Fabricates silently
Style and format control	Limited to prompting	Strong
Upfront effort	Pipeline and index	Data curation and GPU runs

Where Fine-Tuning Earns Its Cost

Fine-tuning is the right tool when the target is behavior rather than knowledge: a consistent tone, a strict output schema, a fixed classification taxonomy, a domain dialect the base model handles poorly. It also makes smaller self-hosted models viable — Llama 2 (Meta, 18 July 2023) ships in 7B, 13B and 70B variants under a license that permits commercial use, and LoRA-style tuning adapts it on modest hardware.

Note the current API reality: as of this writing, OpenAI's hosted fine-tuning covers only the original GPT-3 base models (ada, babbage, curie, davinci) — not gpt-3.5-turbo, not GPT-4. And the approaches combine: nothing prevents a fine-tuned model from serving as the generator inside a RAG pipeline. Lewis et al. fine-tuned their generator too.

Outlook From July 2023

Context windows are growing fast: GPT-4 offers a 32K variant, and Anthropic's Claude has accepted 100,000 tokens since May 2023. We do not expect long context to make retrieval obsolete. Cost scales with tokens processed, and selecting the relevant thousand tokens out of millions remains a search problem regardless of window size. Retrieval is the index; the context window is only the desk.

Three expectations from where we stand in July 2023. Hosted fine-tuning of current chat models will arrive — and still not replace retrieval for facts. Vector search will become a database feature rather than a database category; pgvector points the way. And evaluation, not modeling, will become the bottleneck: measuring retrieval quality on your own corpus. Our default recommendation stands: retrieval first, fine-tune what remains.