Why RAG, not fine-tuning
People reach for fine-tuning to 'teach the model our data' — usually the wrong tool. Fine-tuning changes style and behavior; it's a poor and expensive way to inject facts, and it goes stale the moment your data changes. RAG keeps facts in a store you can update anytime and injects only what's relevant per question. Your knowledge base changes hourly; your model shouldn't have to be retrained to keep up.
The pipeline: chunk, embed, store, retrieve, answer
- Chunk your documents into passages small enough to be specific but large enough to keep context.
- Embed each chunk into a vector with an embedding model and store it in a vector database (pgvector, Pinecone, Qdrant).
- At query time, embed the question, retrieve the nearest chunks, and pass them to the model as grounding context.
- The model answers from the retrieved passages — and cites them, so answers are checkable.
// Retrieve, then ground the answer
const qVec = await embed(question);
const hits = await db.query(
`SELECT text, source FROM chunks
ORDER BY embedding <=> $1 LIMIT 6`, [qVec]); // pgvector cosine distance
const context = hits.map((h, i) => `[${i + 1}] ${h.text}`).join('\n\n');
const answer = await llm.chat([
{ role: 'system', content: 'Answer ONLY from the context. If it is not there, say you do not know. Cite sources like [1].' },
{ role: 'user', content: `Context:\n${context}\n\nQuestion: ${question}` },
]);Retrieval quality is the whole game
If retrieval returns the wrong passages, no model can save the answer. The levers that actually move accuracy: chunk size and overlap (too big buries the answer, too small loses context), hybrid search (combine vector similarity with keyword/BM25 so exact terms like product names aren't missed), and re-ranking the top candidates before you hand them to the model. Most 'the AI is dumb' complaints are really 'the retrieval is bad'.
Instruct the model to answer only from the retrieved context and to say 'I don't know' when it isn't there. That single system-prompt rule is the difference between a bot that cites your docs and one that invents policy.
Multi-tenant RAG: isolate the vectors
In a SaaS, tenant A must never retrieve tenant B's chunks. Scope every query by tenant id (a filter in the vector search, or a per-tenant namespace/collection) — the same data-isolation discipline as any multi-tenant store, applied to embeddings. A leak here isn't a bad answer; it's a data breach.
Key takeaways
- Use RAG to inject facts; use fine-tuning only for style/behavior, not knowledge.
- The pipeline is chunk → embed → store → retrieve → grounded answer with citations.
- Retrieval quality (chunking, hybrid search, re-ranking) decides accuracy — not the model.
- Ground answers strictly in context, and isolate vectors per tenant in multi-tenant SaaS.