AI Implementation
RAG for Consulting: A Plain Guide for Founders.
RAG is a way to make an AI answer from your own files. It stops the AI guessing. For a consulting business, it means your frameworks, notes and past work become something your team can ask questions of.
This guide explains RAG in simple words. It shows what to feed it, how to keep client data safe and how to check it works.
TL;DR
- RAG stands for retrieval-augmented generation. The AI looks up your documents first, then writes an answer.
- It suits question-and-answer work over your own material: frameworks, proposals, playbooks, research.
- Clean files matter more than clever tools. Remove old versions and client secrets before you start.
- Set access rules and test answers on real questions before your team relies on it.
- If you are a $1M+ founder, we can build this with you.
What RAG is, in kitchen terms
Think of a chef who knows a lot. But he has never seen your recipe book.
Ask him for your signature dish and he will guess. It may taste fine. It will not be yours.
RAG hands him your recipe book first. He reads the right page. Then he cooks.
That is the whole idea. The term comes from a 2020 research paper by Lewis and colleagues. IBM Research describes RAG as grounding the model on external sources of knowledge so its answers are better. It has two phases:
- Retrieval. Software finds the snippets that match the question.
- Generation. The AI writes an answer using those snippets.
AWS adds a useful point. RAG lets a model use your organisation's own knowledge without retraining the model. You do not rebuild the AI. You point it at your files.
How it works, step by step
Here is the flow in four steps, following InfoWorld's step-by-step description:
- Ingestion. Your documents are loaded into a special store. It is called a Vector databaseNoun · glossaryA vector database stores text as numerical representations of meaning, so you can search by what something means rather than the exact words it uses.Read the full entry → (a store that finds text by meaning, not just exact words).
- Question. Someone asks in plain language.
- Retrieval. The system finds the pieces of your files closest in meaning to the question.
- Answer. The AI writes a reply using those pieces.
NVIDIA describes the same flow as extraction, retrieval and generation. It also notes that retrieval can mix meaning-based search with keyword search.
Want to see how this works for a whole company? Read our piece on a company knowledge chatbot. To plan how your files should be organised first, see knowledge architecture for AI.
Why consultants and educators care
You sell judgement. That judgement lives in your head, your slides and your old projects.
The pain is familiar:
- Your team asks you the same questions every week.
- A junior hunts through folders for the right template.
- You answer a client query you already answered last year.
RAG helps with all three. Your team asks the system. The system finds your own material and answers from it.
There is a big-firm example. SAP built a tool for its consultants that uses the RAG pattern. AWS describes it giving answers from SAP learning material and notes, with built-in citations. Every answer points back to its source. That is the standard to aim for.
You do not need a giant firm to do this. You need good files and a clear scope.
What to put in, and what to leave out
Not every file belongs. Here is a simple sort.
Good fits
- Your core frameworks and methods
- Playbooks and standard operating procedures (written step-by-step instructions)
- Proposal and report templates
- Training material and course content
- Anonymised case studies
- Answers to questions clients ask often
Poor fits
- Old versions of your frameworks
- Old pricing and retired services
- Contradicting drafts
- Anything with client names or NDA-covered details
- Personal data
One guide for consulting firms says the same. It advises purging superseded framework versions, old pricing and contradictory drafts. It also advises a confidentiality screen before upload.
Think of hiring a new staff member. You would not hand them every file from ten years ago. You would give them the current handbook. Do the same for the AI.
Not sure what counts as your knowledge base? Our guide on what a consultant knowledge base is lays it out.
How to set it up
Here is a plain order of work.
- Pick one job. For example: "Answer team questions about our delivery method." One job beats ten.
- Gather the files. Collect only what supports that job.
- Clean them. Remove old, wrong and secret items.
- Split them into chunks. Chunks are small pieces of text. The AI finds a chunk, not a whole book.
- Store them. This is the vector database step. Our piece on vector database benchmarks shows what to measure first.
- Write the instructions. Tell the AI to answer only from the retrieved pieces and to show its source.
- Test it. More on that below.
Anthropic adds one trick for the chunking step. Its Contextual Retrieval method prepends a short note to each chunk, saying where it came from, before the chunk is stored. A chunk that says "revenue grew 3%" means little alone. With context, the search can find it.
Microsoft's design guide covers a similar path. It lists preparation, chunking, chunk enrichment, embedding, retrieval and evaluation as phases. You can skip the jargon. The order is what counts: prepare, split, store, retrieve, check.
Keep client information safe
This is where consultants must be careful. Your files may hold client secrets. A badly set up system could show the wrong person the wrong file.
OWASP is a respected security body. Its RAG guidance advises storing access rules with every chunk and checking permissions when the system retrieves. It also advises logging each retrieval with who asked.
In plain words:
- Label each file with who may see it.
- Check that label every time someone asks.
- Keep a log, so you can see who saw what.
Okta adds more steps. It recommends tagging content by sensitivity and filtering out personal data before it is embedded.
Think of an office with locked cabinets. The AI is a clerk. The clerk should only open cabinets the asker has a key for.
If client contracts limit how you use their material, talk to your lawyer before you upload it. The safest start is your own IP and anonymised work.
How to test that it works
Do not trust a demo. Test it on real questions.
Make a list of 30 to 50 questions your team really asks. Write the right answer for each. Then run them.
Microsoft suggests checking answers for groundedness, completeness, utilisation, relevancy and correctness. Simple version:
- Grounded. Does the answer come from your files?
- Complete. Does it cover the whole question?
- Relevant. Does it answer what was asked?
- Correct. Is it true?
Red Hat's enterprise guide lists four checks too: context recall, context precision, answer faithfulness and answer relevance. The first two test the search step. The last two test the writing step.
This split is useful. If the answer is wrong, find out which step failed. Did it fetch the wrong page? Or did it fetch the right page and write badly? The fix differs.
The TREC 2024 RAG evaluation checks whether each sentence is supported by the sources it cites. You can copy that habit. Click each citation. Does it back the claim?
When answers are weak, Microsoft says to check that the chunks are right, review the prompt and consider more grounding data. Start with the files. They are usually the problem.
Where RAG falls short, and what to pair it with
RAG is good at one thing: answering questions from your documents. Use it for that.
For other jobs, you may pair it with other methods. AWS Prescriptive Guidance says RAG does not work well when summarising whole documents. It suggests starting with RAG for question-answering and using Fine-tuningNoun · glossaryFine-tuning is further training an existing AI model on your own examples so it adopts a particular style, format, or task.Read the full entry → (extra training of the model) for added tasks like summarising.
So the order is simple. Start with RAG. Add more when you have a clear need.
We cover the choice in RAG vs fine-tuning.
Our view
RAG alone is a filing cabinet with a good clerk. It finds things. It does not think like you.
That is the gap we see with founders. They add a chat tool over their files. Then they find their team still asks them what to do. The tool knows the facts. It does not know the founder's judgement.
So we build more than search. We pull out how you decide. We write it down. Then we build, with Claude Code, a team of AI agents (software helpers, each with one job) that work from your IP and follow your rules. That team is your AI operating systemNoun · glossaryAn AI operating system is the shared layer a business runs its AI on: the models, the packaged know-how, the memory, the live connections to real tools, and the jobs that run on a clock.Read the full entry →.
RAG is one part of that. It gives the agents your facts. The rest gives them your standards. Njin, our sister brand, explains the idea in what an AI operating system for business is.
We start small and build fast. Our aim is a working system live in 30 days. You own it. There is no lock-in.
That is how a founder gets hours back without the quality dropping.
James Killick
Work with us
We built our approach for founder-led consulting and education firms earning $1M+. You have strong IP. You want your team to deliver to your standard without you in every decision.
The AI Orchestrator OS is done with you, not just for you. We build the system with your team and put it in place over six weeks of 1 on 1 setup. It costs $30,000, or 3 x $11,000. 1 on 1 coaching after that starts at $1,950 a month.
Prefer to build yourself? Our platform costs $90 a month. Coaches can also see how we work with coaches.
A simple first week
You can start without us. Here is a small plan.
Day 1. Pick one job for the system. Write it in one sentence.
Day 2. List the ten questions your team asks you most.
Day 3. Find the files that answer them.
Day 4. Remove old versions, client names and personal data.
Day 5. Write the right answer for each question. This is your test sheet.
By Friday you have a clean set of files and a way to score the system. That alone puts you ahead of most.
Plan for the dull parts too. One report estimates that 30 to 50 percent of project cost goes into access control, connectors, logging and data cleaning. Treat that as a rough guide, not a rule. It does show where the effort goes. It is rarely the AI itself.
Your next step
Want a second pair of eyes on which files to start with? Book a strategy call and we will go through your material with you.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv
- Introducing Contextual Retrieval, Anthropic
- What is retrieval-augmented generation (RAG)? IBM Research
- What is RAG? AWS
- Retrieval-augmented generation, step by step. InfoWorld
- What is Retrieval-Augmented Generation? NVIDIA
- Accelerating SAP consulting projects with SAP Joule. AWS for SAP
- RAG for Consulting Firms: AI Knowledge Base Guide
- Design and Develop a RAG Solution on Azure. Microsoft Learn
- RAG Security Cheat Sheet. OWASP
- How to Secure RAG Workflows. Okta
- RAG evaluation phase. Microsoft Learn
- Engineering RAG for the enterprise. Red Hat
- TREC 2024 RAG Evaluation Overview
- Retrieval-Augmented Generation for Specialised Domains. DataDeep.tech
- Comparing RAG and fine-tuning. AWS Prescriptive Guidance
Frequently asked questions.
James Killick
Founder
The AI Orchestrator. 10+ years building digital products and 200+ apps shipped, now helping $1M+ educators and consultants turn their IP into AI-powered delivery systems.
James Killick founded and runs The AI Orchestrators.
More from James Killick