Skip to content

AI Implementation

RAG for Consulting: A Plain Guide for Founders.

JK
9 min read

RAG is a way to make an AI answer from your own files. It stops the AI guessing. For a consulting business, it means your frameworks, notes and past work become something your team can ask questions of.

This guide explains RAG in simple words. It shows what to feed it, how to keep client data safe and how to check it works.

TL;DR

  • RAG stands for retrieval-augmented generation. The AI looks up your documents first, then writes an answer.
  • It suits question-and-answer work over your own material: frameworks, proposals, playbooks, research.
  • Clean files matter more than clever tools. Remove old versions and client secrets before you start.
  • Set access rules and test answers on real questions before your team relies on it.
  • If you are a $1M+ founder, we can build this with you.

What RAG is, in kitchen terms

Think of a chef who knows a lot. But he has never seen your recipe book.

Ask him for your signature dish and he will guess. It may taste fine. It will not be yours.

RAG hands him your recipe book first. He reads the right page. Then he cooks.

That is the whole idea. The term comes from a 2020 research paper by Lewis and colleagues. IBM Research describes RAG as grounding the model on external sources of knowledge so its answers are better. It has two phases:

  1. Retrieval. Software finds the snippets that match the question.
  2. Generation. The AI writes an answer using those snippets.

AWS adds a useful point. RAG lets a model use your organisation's own knowledge without retraining the model. You do not rebuild the AI. You point it at your files.

How it works, step by step

Here is the flow in four steps, following InfoWorld's step-by-step description:

  1. Ingestion. Your documents are loaded into a special store. It is called a (a store that finds text by meaning, not just exact words).
  2. Question. Someone asks in plain language.
  3. Retrieval. The system finds the pieces of your files closest in meaning to the question.
  4. Answer. The AI writes a reply using those pieces.

NVIDIA describes the same flow as extraction, retrieval and generation. It also notes that retrieval can mix meaning-based search with keyword search.

Want to see how this works for a whole company? Read our piece on a company knowledge chatbot. To plan how your files should be organised first, see knowledge architecture for AI.

Why consultants and educators care

You sell judgement. That judgement lives in your head, your slides and your old projects.

The pain is familiar:

  • Your team asks you the same questions every week.
  • A junior hunts through folders for the right template.
  • You answer a client query you already answered last year.

RAG helps with all three. Your team asks the system. The system finds your own material and answers from it.

There is a big-firm example. SAP built a tool for its consultants that uses the RAG pattern. AWS describes it giving answers from SAP learning material and notes, with built-in citations. Every answer points back to its source. That is the standard to aim for.

You do not need a giant firm to do this. You need good files and a clear scope.

What to put in, and what to leave out

Not every file belongs. Here is a simple sort.

Good fits

  • Your core frameworks and methods
  • Playbooks and standard operating procedures (written step-by-step instructions)
  • Proposal and report templates
  • Training material and course content
  • Anonymised case studies
  • Answers to questions clients ask often

Poor fits

  • Old versions of your frameworks
  • Old pricing and retired services
  • Contradicting drafts
  • Anything with client names or NDA-covered details
  • Personal data

One guide for consulting firms says the same. It advises purging superseded framework versions, old pricing and contradictory drafts. It also advises a confidentiality screen before upload.

Think of hiring a new staff member. You would not hand them every file from ten years ago. You would give them the current handbook. Do the same for the AI.

Not sure what counts as your knowledge base? Our guide on what a consultant knowledge base is lays it out.

How to set it up

Here is a plain order of work.

  1. Pick one job. For example: "Answer team questions about our delivery method." One job beats ten.
  2. Gather the files. Collect only what supports that job.
  3. Clean them. Remove old, wrong and secret items.
  4. Split them into chunks. Chunks are small pieces of text. The AI finds a chunk, not a whole book.
  5. Store them. This is the vector database step. Our piece on vector database benchmarks shows what to measure first.
  6. Write the instructions. Tell the AI to answer only from the retrieved pieces and to show its source.
  7. Test it. More on that below.

Anthropic adds one trick for the chunking step. Its Contextual Retrieval method prepends a short note to each chunk, saying where it came from, before the chunk is stored. A chunk that says "revenue grew 3%" means little alone. With context, the search can find it.

Microsoft's design guide covers a similar path. It lists preparation, chunking, chunk enrichment, embedding, retrieval and evaluation as phases. You can skip the jargon. The order is what counts: prepare, split, store, retrieve, check.

Keep client information safe

This is where consultants must be careful. Your files may hold client secrets. A badly set up system could show the wrong person the wrong file.

OWASP is a respected security body. Its RAG guidance advises storing access rules with every chunk and checking permissions when the system retrieves. It also advises logging each retrieval with who asked.

In plain words:

  • Label each file with who may see it.
  • Check that label every time someone asks.
  • Keep a log, so you can see who saw what.

Okta adds more steps. It recommends tagging content by sensitivity and filtering out personal data before it is embedded.

Think of an office with locked cabinets. The AI is a clerk. The clerk should only open cabinets the asker has a key for.

If client contracts limit how you use their material, talk to your lawyer before you upload it. The safest start is your own IP and anonymised work.

How to test that it works

Do not trust a demo. Test it on real questions.

Make a list of 30 to 50 questions your team really asks. Write the right answer for each. Then run them.

Microsoft suggests checking answers for groundedness, completeness, utilisation, relevancy and correctness. Simple version:

  • Grounded. Does the answer come from your files?
  • Complete. Does it cover the whole question?
  • Relevant. Does it answer what was asked?
  • Correct. Is it true?

Red Hat's enterprise guide lists four checks too: context recall, context precision, answer faithfulness and answer relevance. The first two test the search step. The last two test the writing step.

This split is useful. If the answer is wrong, find out which step failed. Did it fetch the wrong page? Or did it fetch the right page and write badly? The fix differs.

The TREC 2024 RAG evaluation checks whether each sentence is supported by the sources it cites. You can copy that habit. Click each citation. Does it back the claim?

When answers are weak, Microsoft says to check that the chunks are right, review the prompt and consider more grounding data. Start with the files. They are usually the problem.

Where RAG falls short, and what to pair it with

RAG is good at one thing: answering questions from your documents. Use it for that.

For other jobs, you may pair it with other methods. AWS Prescriptive Guidance says RAG does not work well when summarising whole documents. It suggests starting with RAG for question-answering and using (extra training of the model) for added tasks like summarising.

So the order is simple. Start with RAG. Add more when you have a clear need.

We cover the choice in RAG vs fine-tuning.

Our view

RAG alone is a filing cabinet with a good clerk. It finds things. It does not think like you.

That is the gap we see with founders. They add a chat tool over their files. Then they find their team still asks them what to do. The tool knows the facts. It does not know the founder's judgement.

So we build more than search. We pull out how you decide. We write it down. Then we build, with Claude Code, a team of AI agents (software helpers, each with one job) that work from your IP and follow your rules. That team is your .

RAG is one part of that. It gives the agents your facts. The rest gives them your standards. Njin, our sister brand, explains the idea in what an AI operating system for business is.

We start small and build fast. Our aim is a working system live in 30 days. You own it. There is no lock-in.

That is how a founder gets hours back without the quality dropping.

James Killick

Work with us

We built our approach for founder-led consulting and education firms earning $1M+. You have strong IP. You want your team to deliver to your standard without you in every decision.

The AI Orchestrator OS is done with you, not just for you. We build the system with your team and put it in place over six weeks of 1 on 1 setup. It costs $30,000, or 3 x $11,000. 1 on 1 coaching after that starts at $1,950 a month.

Prefer to build yourself? Our platform costs $90 a month. Coaches can also see how we work with coaches.

A simple first week

You can start without us. Here is a small plan.

Day 1. Pick one job for the system. Write it in one sentence.

Day 2. List the ten questions your team asks you most.

Day 3. Find the files that answer them.

Day 4. Remove old versions, client names and personal data.

Day 5. Write the right answer for each question. This is your test sheet.

By Friday you have a clean set of files and a way to score the system. That alone puts you ahead of most.

Plan for the dull parts too. One report estimates that 30 to 50 percent of project cost goes into access control, connectors, logging and data cleaning. Treat that as a rough guide, not a rule. It does show where the effort goes. It is rarely the AI itself.

Your next step

Want a second pair of eyes on which files to start with? Book a strategy call and we will go through your material with you.

Sources

FAQ

Frequently asked questions.

JK

James Killick

Founder

The AI Orchestrator. 10+ years building digital products and 200+ apps shipped, now helping $1M+ educators and consultants turn their IP into AI-powered delivery systems.

James Killick founded and runs The AI Orchestrators.

Keep reading

Ready to find out where your biggest AI opportunity is?

Take the assessment. It takes about 5 minutes. You'll get a clear picture of how ready your business is.