AI Tools
AI Diagnostic Assessment: Build One From Your Own Expertise.
An AI diagnostic assessment is a short test that finds what a person knows, what they get wrong and why. AI helps you write it, mark it and spot patterns fast. But the quality comes from your expertise, not the tool.
If you run a coaching, course or consulting business, this matters. Your method is the asset. A diagnostic built from your method is the fastest way to show new clients where they stand. It also lets your team do the first read without you.
This guide shows how to build one, check it and use the results. It is written for founders and educators, not doctors. (Many search results for this phrase cover medical scans. That is a different topic.)
TL;DR
- A diagnostic assessment finds gaps and misconceptions. It does not just give a score.
- AI can draft the questions in minutes. You must feed it your learning goals and your method.
- Always review the output. Studies and university guides report that AI questions can be misaligned or unclear.
- Read results as patterns of strengths and needs. Do not use them to label people.
- If the real goal is to get your judgment out of your head and into a system, that is a bigger build. We cover it below.
What a diagnostic assessment actually is
Think of a doctor's check-up, not an exam. An exam says pass or fail. A check-up says what is wrong and what to do next.
The Michigan Assessment Consortium defines diagnostic assessment as a way to find understanding, misconceptions and knowledge gaps. It is not only a check on whether someone can do a task.
That is the key difference. A good diagnostic tells you why the person got it wrong.
A simple example
The same guide uses fractions. Instead of one question, it checks the small skills inside:
- Naming the numerator and denominator (the top and bottom numbers)
- Finding a common denominator
- Making equivalent fractions
- Adding the top numbers only once the bottoms match
- Simplifying the answer
A learner who fails the whole task might only be stuck on one of these. You fix that one thing. You do not repeat the whole lesson.
Why the idea works outside school
Your clients are the same. A client who "can't scale" is rarely stuck everywhere. They are stuck on one or two steps of your method.
A diagnostic finds those steps. That saves your time and theirs.
Where diagnostics fit in your business
Most founders with a method use diagnostics in four places:
- Before a sale. A short quiz shows a prospect where they stand. It also shows why they need your help.
- At onboarding. You place each client at the right starting point.
- Mid-program. You see who is stuck, and on what.
- After delivery. You show progress with proof.
The education research agrees on the timing. The National Center on Intensive Intervention explains that diagnostic data come in after progress checks show a learner is not responding well enough. They then help you decide what to change.
So a diagnostic is a second step. First you see that something is off. Then you dig into why.
How to build one with AI, step by step
Think of AI as a new kitchen hand. It chops fast. But it does not know your recipe unless you hand it over.
Step 1: Write down what a learner must be able to do
Start with goals, not questions. Broad goals are too fuzzy to test.
The 2025 Kappan article suggests unpacking broad standards into specific, measurable targets before you draft any questions.
So "understands pricing" becomes:
- Can name three pricing models
- Can pick the right one for a given client
- Can explain the trade-off in one paragraph
Now each target can be tested.
Step 2: Plan coverage in a simple table
The same Kappan piece suggests a table of specifications. That is a grid. One side lists your goals. The other side lists how hard each question is and what you have already taught.
It shows gaps. Maybe you have six questions on pricing and none on delivery. A glance at the grid tells you.
Step 3: Give AI the full brief
Vague prompts give vague tests. The University of Maryland's assessment generator prompt asks you to state five things first:
- Learning objectives
- The learner's level
- The purpose of the assessment
- Time limits
- Any accommodation needs
It also advises setting reading levels, language needs, question types and how you will analyse the data.
For a business, add two more:
- Your own framework, in your own words
- Three or four real examples of a good answer and a bad one
The second item matters most. It teaches the AI what "good" means in your world.
Step 4: Ask for questions that find misconceptions
A good diagnostic question has wrong answers that look right. These are called distractors (wrong options that tempt people).
Indiana University's teaching page gives an example prompt that asks for multiple-choice questions with plausible distractors, an answer key and feedback for right and wrong choices.
Each wrong option should match a real mistake. Then a wrong pick tells you something.
Try this recipe:
- One question per skill
- Three wrong options, each tied to a common mistake
- A one-line note on why each wrong option is wrong
- A tag showing which skill the question tests
The tag is the secret. It lets you sort results by skill later.
Step 5: Make AI check its own work
The University of Illinois Chicago guide suggests giving the AI your directions, objectives and marking criteria. Then ask it to map each objective to the question wording and flag anything unclear or missing.
It is a cheap first pass. It does not replace your eyes.
Step 6: Review it yourself
This step cannot be skipped. The Kappan authors report that they found AI questions that were misaligned, unsuitable for their students or filed under the wrong evidence type. Teacher review fixed it.
The University of Pennsylvania's teaching centre says the same. AI can build question banks and practice tests, but instructors should review, evaluate and edit the output.
One more option comes from research. A 2024 paper in Frontiers in Education looks at tying questions to evidence statements. Humans then judge whether each generated question fits.
A quick quality check before you use it
Run every question through this list. It blends advice from two university guides.
| Check | Ask yourself |
|---|---|
| Purpose | Does this question test the skill I meant? |
| Level | Is it the right difficulty for this learner? |
| Clarity | Are the instructions plain? Is there only one fair reading? |
| Facts | Is every claim and technical detail true? |
| Access | Is the language inclusive? Can everyone use it? |
| Context | Does it still fit my method and my clients? |
The first four rows echo Xavier University's AI assessment design quick guide, which stresses objective fit, course level, clear instructions and accessible language. The facts, access and context rows come from the Illinois Chicago final review list.
Mind privacy
Before you paste anything in, think about who owns the data. Penn's guide tells educators to consider student privacy and institutional AI policies first.
For a business, the same logic applies. Do not paste client names or private answers into a tool you have not checked. Strip them out first.
How to read the results
A score alone is thin. Look at the pattern of mistakes.
Vanderbilt's IRIS Center recommends examining error patterns in work samples to guess which specific skill is weak. It says this is better than relying on an overall score.
Here is how that looks in practice:
- Ten people miss question 4. Check the question first. It may be unclear.
- One person misses all pricing questions. That is a skill gap. Teach pricing.
- Everyone misses the same wrong option. A shared myth. Fix it in your content.
Do not label people
The Georgia Department of Education's K-2 toolkit says to read results as patterns of strengths and needs. Use them to decide what to teach first. Do not use them to stick a label on someone.
It is written for early-grade teachers. The idea carries over. "Needs work on delivery" helps. "Weak client" does not.
Keep a human in the loop
AI can read a lot of answers quickly. But it can be fooled.
The Massachusetts public higher-education GenAI assessment guidebook warns that AI may spot surface patterns instead of the thing you meant to measure. It says AI analysis should be checked against expert human judgment, because context still needs a person to read it.
In plain words: the AI is a fast reader. You are the one who knows what the answers mean.
Use the diagnostic to attract clients
A good diagnostic can also be a lead tool. A prospect takes a ten-minute quiz. They get a result that names their gap. Then you offer the fix.
Two tips:
- Keep it short. Ten to fifteen questions is plenty.
- Show a result that is useful on its own. People trust what helps them for free.
Once the page is live, it helps to see whether AI search shows it. A tool to check your AI Overview visibility can show how your page appears in Google's AI answers.
The bigger job: put your judgment into a system
A diagnostic quiz is a good start. But the quiz only asks questions. Someone still has to read the answers, decide what is wrong and say what to do next.
For most founders, that someone is you. That is the bottleneck.
The real value is not the quiz. It is your judgment behind it. Which gap matters most? What do you tell this client? What is the next step? If that stays in your head, every diagnostic ends up on your desk.
Here is how we handle it. We write your method down as rules, then build AI employees in Claude Code that apply those rules the way you would. One scores the answers. One picks the gap that matters most. One drafts the next-step note for your review. You stay in the loop on the calls that need you. Your team handles the rest.
If you want to go deeper, read our founder's guide on AI-assisted diagnosis in consulting. For a quick read on where your business stands with AI, take the free AI readiness assessment. Our piece on the five levels of AI maturity explains what each score means.
How we can help
A quiz is the first brick. The AI Orchestrator OS is the house. It is an always-on AI operating systemNoun · glossaryAn AI operating system is the shared layer a business runs its AI on: the models, the packaged know-how, the memory, the live connections to real tools, and the jobs that run on a clock.Read the full entry → built from your own method and shared across your coaches and consultants. Six weeks of 1 on 1 setup coaching gets it running, and your team learns to use it.
The AI Orchestrator OS costs $30,000, or 3 x $11,000. After setup, 1 on 1 coaching runs from $1,950 a month. The platform is free for 12 months with the OS. See how the AI Orchestrator OS works, or look at the platform.
We work with founder-led education and consulting businesses earning $1M or more a year. If that is you, the pages for coaches and online educators show what we do for each.
Related reading
If you are mapping out the wider picture, these two guides help:
Sources
- Learning Point: What are "diagnostic assessments"? (Michigan Assessment Consortium)
- Using Diagnostic Data to Inform Intervention Planning (National Center on Intensive Intervention)
- Page 5: Diagnostic Assessment (Vanderbilt IRIS Center)
- Collaborate with generative AI to improve classroom assessments (Kappan)
- Assessment Generator (AI Resources, University of Maryland)
- Generative AI in teaching and learning (Teaching.IU, Indiana University)
- Using Google AI to Support Assessment Design (University of Illinois Chicago)
- Using AI to Create Assessments (Penn CETLI)
- Developing valid assessments in the era of generative AI (Frontiers in Education)
- AI Assessment Design Quick Guide (Xavier University)
- K-2 Diagnostic Assessment Toolkit (Georgia Department of Education)
- GenAI Assessment Guidebook (Massachusetts Department of Higher Education)
- AI Overview Checker (BabyLoveGrowth)
Your next step: take the free AI readiness assessment and see your score.
Frequently asked questions.
James Killick
Founder
The AI Orchestrator. 10+ years building digital products and 200+ apps shipped, now helping $1M+ educators and consultants turn their IP into AI-powered delivery systems.
James Killick founded and runs The AI Orchestrators.
More from James Killick