RAG for Company Knowledge: A Plain Guide.

RAG lets an AI answer questions from your own documents. It looks up the right pages first. Then it writes the answer from those pages and shows where it found them.

Think of a new hire with a good filing cabinet. They don't guess. They open the cabinet, find the page and read it back to you. That is RAG (retrieval-augmented generation, which just means "look it up, then answer").

This guide shows how it works, what to prepare, where it breaks and how to start.

TL;DR

• RAG makes an AI answer from your files, not from memory. Stanford HAI describes it as a model looking up facts from an outside source first.
• Your answers are only as good as your documents. Old or messy files give old or messy answers.
• Most of the work is not the AI. It is cleaning files, setting who can see what, and testing.
• Set permissions before launch. Test with real questions. Keep a person in charge of the content.
• RAG pays off most when people ask the same kind of question again and again.

What RAG for company knowledge does, in kitchen terms

Picture a chef who has read every cookbook once. Ask for your gran's lasagne and the chef guesses. The result sounds right. It might not be.

Now give the chef your gran's recipe card. Ask again. The chef reads the card and cooks from it.

That card is RAG. The chef is the language model (the AI that writes text). The recipe box is your company knowledge.

McKinsey's explainer describes the steps like this:

1. The company loads its documents and indexes them. Often it turns chunks of text into embeddings (number codes that capture meaning).
2. A person asks a question.
3. The system finds the passages that match.
4. The model writes an answer from those passages.

That is it. No magic. A search step, then a writing step.

What company knowledge fits RAG

RAG works well when the answer already sits in a document. Typical fits, per McKinsey, are internal knowledge chatbots, customer service chatbots and drafting helpers.

Good examples:

• Policies and handbooks
• Course materials and lesson notes
• Client playbooks and frameworks
• Support answers and past tickets
• Proposals, call notes and templates

If your team asks "where is that doc?" ten times a week, RAG is a fit.

What RAG does badly

It is not a cure for everything. Be honest about the limits.

• Old files give old answers. McKinsey notes that inaccurate or outdated source data leads to wrong outputs.
• Charts, images and complex slides are hard. The same explainer says these can trip it up.
• It still can make things up. TechTarget explains that RAG reduces made-up answers but does not remove them. If the lookup misses a page, the model may fill the gap with a believable guess.
• Reasoning is a different job. A Microsoft Research survey argues no single method fits every task. Looking up a fact is not the same as working through a problem.

So use RAG for "what does our policy say?" It is weaker for "what should we do about this odd case?" That second one needs judgment, and we cover it below.

The parts you need

A RAG system has a few simple parts. Think of a small library.

Part	Everyday picture	What it does
Loader	The person who collects books	Pulls files from drives, wikis and tools
Cleaner	The person who removes junk pages	Extracts plain text and strips clutter
Chunker	Cutting a book into index cards	Splits text into small pieces
Embedder	The librarian who tags each card	Turns each piece into a number code
Vector database	The card catalogue	Stores the codes so matches can be found fast
Retriever	The person fetching cards	Finds the best pieces for a question
Language model	The writer	Writes the answer from those pieces
Checker	The editor	Adds sources, filters and tests

Anyscale's RAG docs describe the same flow: load, extract and clean, chunk, embed, store. They also make a point worth remembering. Retrieval can't recover information that was never pulled out, or that was cut up badly.

In plain words: if the card is torn, the librarian can't find it.

We wrote a separate guide on picking the card catalogue: five benchmarks for a vector database. And if you are unsure whether to use RAG at all, see RAG versus fine-tuning. For a step-by-step build, DevWiz has a walkthrough on how to build a RAG pipeline.

How to prepare your documents

This is where most projects win or lose. The AI is the easy bit.

Step 1: Pick one job first

Don't load everything. Pick one question type. For example: "What do I tell a client who asks for a refund?"

One job. One set of files. One owner.

Step 2: Gather the best files

Collect the documents people actually trust. Leave out drafts, duplicates and anything nobody has opened in years.

A small, clean pile beats a big, messy one.

Step 3: Fix the mess

• Remove old versions. Keep one current copy.
• Give each file a clear title and date.
• Turn scanned pages into real text.
• Describe charts in words next to the chart.
• Mark who may see each file.

That last point matters more than people expect. We cover it next.

Step 4: Cut sensibly

Chunks (the small pieces of text) should hold one idea each. Cut mid-sentence and the meaning breaks. Cut a table in half and the numbers lose their labels.

Test a few cuts by hand. Read them. Ask: would a person understand this card alone?

Step 5: Name an owner

Somebody must own the content. Not "the team". One person with a name.

Keep company data safe

A RAG system can read a lot. That is the point. It is also the risk.

The OWASP RAG Security Cheat Sheet says RAG adds new weak spots at every stage of the pipeline. Examples include poisoned documents, hidden instructions inside retrieved text and getting around access rules.

It also lists basic controls:

• Put access labels on every chunk
• Keep each client's data apart
• Protect what goes into the model's context
• Fail closed, which means if a check breaks, the system refuses and doesn't guess

Here is the kitchen version. Your staff canteen has a locked store room. A good cook only fetches from shelves they have a key for. RAG needs the same keys.

A simple permissions test

1. Make two test users. One sees everything. One sees little.
2. Ask both the same sensitive question.
3. The second user must get a refusal, not a partial answer.

If it fails, don't launch.

Health and personal data in the US

If your files hold patient information, take extra care. A JAMA Network article on chatbot privacy notes that typing protected health information into a chatbot without a business associate agreement can count as an unauthorised disclosure under HIPAA. Ask your privacy lawyer or compliance officer before any health data goes near an AI tool.

For the wider risk picture, NIST publishes a generative AI profile as part of its AI Risk Management Framework. It is a useful checklist for policies on made-up answers and source tracking.

Governance without the headache

You need simple rules: who may add files, who approves them, who reviews mistakes. A short AI readiness assessment can help you map your workflows before you buy tools.

We also have longer guides on AI system governance, AI data governance and layered AI agent security.

Test before you trust it

A demo that works on three questions proves nothing. Industry research from interviews with 13 practitioners found most real RAG projects were still question-answering prototypes. Data protection, security, quality and file preparation were the main practical worries.

So test like a hard-nosed buyer.

Build a question list

Write 50 to 100 real questions. Pull them from support emails, team chats and client calls. Add the answer a good person would give.

Include the nasty ones:

• Questions with no answer in your files
• Questions where two files disagree
• Questions that ask for private data
• Questions phrased badly

Score two things

Score the lookup and the answer apart. It is like checking the cook and the shopping separately.

Check	Plain question	Pass sign
Lookup	Did it find the right pages?	The right file shows up in the top results
Faithful	Does the answer stick to those pages?	Every claim matches a source
Complete	Did it miss anything key?	Nothing important is left out
Refusal	Does it say "I don't know" when it should?	No guessing on empty topics
Access	Does it respect who can see what?	Test users see only their own files
Cost	What does each answer cost?	Within your budget at real volume

Open tools exist to automate scoring, but a person should read a sample every time. Tools can be wrong too.

Set a bar

Decide the pass mark before you test. Not after. A system that is right 60% of the time sounds fine until you picture it advising a client.

Keep it fresh

Company knowledge changes. Prices move. Policies update. A course gets a new module.

If the files change and the index doesn't, the AI keeps quoting the old version. Confidently.

A simple routine:

• Weekly: Re-index changed files. Skim a handful of answers.
• Monthly: Review questions the system failed or refused. Add missing documents.
• Quarterly: Remove dead files. Re-run your test list. Check access labels.
• Always: Let users flag a bad answer in one click.

Each flagged answer is free training data for your content owner. Fix the file, not just the answer.

Where these projects break

The same few things go wrong again and again.

1. Too much, too soon. Teams load every folder. Results turn muddy.
2. No owner. The files rot. Trust drops.
3. No permissions plan. Someone finds a salary sheet in a chat answer.
4. No test list. Nobody knows if it improved or got worse.
5. Wrong job. The team wants judgment and builds a lookup tool.

The last one is the big one for founder-led businesses.

Perspective: lookup is not judgment

We build AI systems for educators and consultants, so this is the gap we think about most.

Much of your value is not in your documents. It is in how you decide. Which client to take. What to say to a stuck student. When to bend a rule.

RAG can fetch your handbook. It can't tell you why you made a call last Tuesday. That reasoning often lives in your head and nowhere else.

So we treat RAG as one tool in the box. It is the filing cabinet, and it is good at that job. But a business that wants to run without its founder needs more than a cabinet. It needs your decision rules written down, tested and shared across a few focused AI helpers, each with one job. We build those with Claude Code, and knowledge architecture is where that work starts.

Our advice, plainly:

• Start with a small RAG tool on one clear job.
• Write down the rules behind your answers, not just the answers.
• Keep a human in the loop for anything that touches money, health or a client's trust.
• Own your system. Don't get locked into a black box.

It isn't flashy, but it holds up.

James Killick

Build or buy

You have three honest options.

Route	Good for	Watch out for
Off-the-shelf chat tool with file upload	Quick tests on a few files	Weak permissions, little control
Build it in-house	Teams with engineers and time	Slow, and upkeep never ends
Work with a specialist	Founders who want it live and owned	Pick someone who shows you the build

For a $1M+ education or consulting business, the question is rarely "can we build a bot?" It is "can the bot answer the way we would?" That second part needs your expertise, captured properly.

How we can help

We work with founder-led education and consulting businesses. Our Foundation Month is $3,500 one-off. It is a focused start. Our 90-day program is $18,500 one-off. We build with you, step by step, so your team can deliver to your standard without you in every decision.

Want to build on your own? Our Platform Access is $90 per month or $900 per year. You can also read about generative AI consulting for knowledge businesses.

A realistic first 30 days

Here is a sensible plan for a first pass.

Week 1. Pick one job. List the 50 questions. Name the content owner.

Week 2. Gather and clean the files. Set access labels. Decide the pass mark.

Week 3. Build the first version. Run your question list. Read the failures.

Week 4. Fix files and settings. Re-test. Release to a small group, with a flag button.

Then grow it. One job at a time.

Your next step

Pick one question your team asks every week. Write it down today. Then book a call about our 90-day program and we will show you how to turn it into a working system.

Sources

• What is RAG? Stanford HAI
• What is RAG? McKinsey
• Understanding the limitations and challenges of RAG systems, TechTarget
• Retrieval Augmented Generation (RAG) and Beyond, Microsoft Research
• RAG basics, Anyscale Docs
• RAG Security Cheat Sheet, OWASP
• Health Care Privacy Risks of AI Chatbots, JAMA Network
• AI Risk Management Framework, NIST
• Retrieval-Augmented Generation in Industry: An Interview Study