Human in the Loop AI: A 90-Day Prototype for Founders

Human in the loop AI means agents do the repeatable work in your business, and people still approve the calls that matter. You keep your standard while volume grows past what you can check yourself. The first move is not buying software. It is mapping where your judgement has to stay human, before anything gets built.

What human in the loop AI means for a founder-led business

Forget the machine-learning meaning of the term. That one is about people labelling data to train a model. This is about how work runs day to day.

In a consultancy or education business, agents handle the repeatable steps. You, or someone you trust, check the parts that carry risk or carry your name.

Think of a kitchen. The AI is the prep cook. It chops, measures and follows the recipe. You are still the head chef, tasting the sauce before it leaves the pass.

Your method is the business. Hand it to AI with no checks and it gets watered down. Keep it locked in your head and you can't grow past your own hours. Human in the loop is the path between the two.

Slalom's research on designing for humans in the loop is blunt about the cost of skipping it. Unexplained AI decisions erode customer confidence, and trust is slow and expensive to rebuild. Good design protects that trust. It also keeps reviewers thinking, instead of clicking approve on autopilot.

What good design gives you:

• Your standard holds when the volume goes up.
• Judgement calls stay with people who understand the stakes.
• Your team gets sharper, because they review good work instead of doing rote work.
• The business runs without you in every decision.

Which workflow should you pilot first?

Don't start with your hardest workflow. Start with the one that teaches you the most for the least risk.

Run each candidate through four checks:

1. Does it repeat? If it runs differently every time, leave it for later.
2. What does a mistake cost? Low stakes need light checks. High stakes need more.
3. How often does it happen? High-volume work gives you data in weeks. Rare work takes months.
4. Do you have examples? Past reports, templates and client work are what the agent learns your standard from.

Good first pilots in education and consulting are client intake, research, the first draft of a report, or recommendations built from a template.

Map where the decision gets made in that workflow. Write down what good looks like before you build anything.

A practical tip. Pick the workflow your team complains about most. It is usually repetitive and high volume, and nobody will miss doing it by hand.

Here is what a good pilot looks like. In a Claude skills workshop we ran, a team's weekly cohort progress report took up to four hours by hand. We built it as a Claude skill on the call, got it right once, and saved it. The same report then ran in 22 minutes. Boring, weekly, repeatable. That is the profile you want.

How to design an agent network that keeps humans in charge

One agent is not enough. You need a system where each agent has one job and hands work on cleanly.

Anthropic calls this the orchestrator-workers pattern. A central model breaks a task down, hands the pieces to worker models, and pulls the results together. The same guide notes that agents can pause for human feedback at checkpoints. In professional services, the orchestration layer is what controls sequencing, approvals and exceptions, and it logs every action along the way.

Think of it like staffing a restaurant. You wouldn't let the dishwasher plate the food. Every role has a lane.

Typical roles:

• Triage agent. Sorts incoming work by type and urgency.
• Extractor agent. Pulls facts and evidence from source material.
• Drafter agent. Writes the first version from your rubrics and templates.
• Decision agent. Flags anything that needs a judgement call.
• Monitor agent. Watches quality and flags drift over time.

Then split the system into two zones. In the autonomous zone, agents get on with low-risk, repeatable work. In the governed zone, nothing moves without a human sign-off.

That split stops one agent quietly making a call it was never trusted to make. Our guide to AI agent orchestration goes deeper on the roles.

What controls keep reviewers effective, not overwhelmed?

Building the agents is half the job. Without the right controls, reviewers either drown in approvals or start rubber-stamping everything.

The rubber-stamp risk is real enough that the EU wrote it into law. Article 14 of the EU AI Act says high-risk AI systems must be built so people can oversee them. It also names "automation bias", the habit of trusting the machine by default, as something those people must watch for.

Three controls matter most.

Confidence-based routing. The agent rates how sure it is. Clear cases move on. Unclear ones pause for a person. Zapier's human-in-the-loop guide sets the same rule: when confidence drops below a threshold you set, the workflow stops and escalates instead of guessing. Start strict. Loosen the threshold only as the agent earns it.

Handover bundles. When an agent passes work to a person, it hands over more than the output. It includes the source data, the reasoning and a note on what changed. That is your audit trail. It turns a black box into something a reviewer can check in minutes.

One review workspace. Reviewers need one place to see what is waiting, not five tools. Fieldguide's audit platform shipped exactly this pairing, a Kanban-style board plus a workspace built for reviewing agent output.

Controls that keep review manageable:

• Start with strict thresholds and loosen them as trust builds.
• Log every handover with sources and reasoning attached.
• Give reviewers one dashboard, not five inboxes.
• Track how long review takes, not just whether it happened.

We built our own AI dashboard around this exact problem. Agents propose work as rows in one queue. A person approves, rejects or changes each one. The model is never given a tool that can approve its own work. For the reviewer workload maths, see our piece on human in the loop workflows.

A 90-day plan to build and pilot your first system

You don't need a year-long transformation project. You need 90 days and a clear plan.

Days 0 to 30: discovery. Map the workflows. Write down your rubrics, your decision rules, and examples of good and bad work. This is the unglamorous part, and skipping it breaks everything after. Pick one pilot and set your baseline: hours spent, error rate and turnaround time.

Days 30 to 60: build and shadow. Build the orchestrator and agents. Run them in shadow mode. The AI does the work quietly alongside your team, but nobody acts on its output yet. You are only comparing. Track how often it gets flagged and why, then tune the thresholds before real work depends on them.

Days 60 to 90: supervised launch. Go live with a person checking everything at first. Adjust thresholds based on real results. Name an owner for each type of decision. Report hours saved and error rates.

Don't skip shadow mode to save time. It is the cheapest 30 days you will spend, because mistakes there cost nothing.

So when does a task run on its own? You count. If it works five times out of ten, work on the other five. Get it to six, then seven, then eight. When it is right nine times out of ten, turn it into a routine and let it run.

By day 90 you should have a working prototype, real numbers, and a clear view of what to scale next.

What goes wrong, and how to fix it

Most human in the loop projects don't fail because of the AI. They fail because of what gets skipped.

Skipping codification. Agents get built before anyone writes down the rubrics. Fix: write the examples, the exceptions and the "how we handle the weird case" notes first. Codelevate's guide to productising a consulting service makes the point well. Most consulting hours are not judgement at all. They are your standard, done by hand, over and over. Capture that standard before you touch a build tool.

No audit trail. Nobody can explain why the AI made a call, so nobody trusts it. Fix: require a handover bundle on every output.

Measuring the wrong thing. You count logins, not results. Fix: track hours saved, error rates and client impact.

Set and forget. The system works for a month, so people stop reading what it produces. Fix: keep a person on the final check, and keep sampling the work even after thresholds loosen.

Quick checklist:

• No written rubric? Stop and write one.
• No record of why a decision was made? Add it now.
• Only tracking usage? Switch to hours saved and error rates.

Training your team for the new setup

Your team's job changes. That is the honest bit most vendors leave out.

They stop doing the repeatable task by hand. They start reviewing, correcting and teaching the system what good looks like.

That takes training, not a memo. Teach reviewers to spot a weak AI draft fast. Show them what a confidence score means. Walk them through the handover bundle so they know where the sources sit.

Practice on low-stakes examples works well. AI role-play training was built for sales and support teams, but the idea carries straight over to reviewers. It gives them a safe place to practise judgement calls before real work depends on them.

Let the person who already does the job do the teaching. If someone has run your inbox or intake for years, they should sit with Claude and correct it one case at a time. Each rule gets saved as a skill as it lands. That is cheaper and more accurate than you trying to explain their job to a model.

Your best reviewers will be the people who already do the task well. They won't always be the ones most excited about AI.

Who is responsible when the AI gets it wrong?

Answer this before launch, not after the first mistake.

Someone is always accountable for a decision. Not "the system". A named person.

Set ownership per decision type. If the AI drafted a recommendation and a reviewer approved it, the reviewer owns that call.

Be open with clients about where AI is involved. Nobody expects you to hand-write every document forever. But letting AI make unchecked judgement calls on work a client pays for is where trust breaks.

Watch for bias from your own examples. If your rubric came from a narrow set of cases, the AI repeats that narrowness at scale. Review your source examples the way you would review a new hire's training.

Technical snags you will hit

The hard part is rarely the AI. It is plugging it into what you already use, like your CRM, your documents and your delivery tools. A system that sits off to the side is a tool nobody opens.

Common snags:

• Messy source data. Past client work scattered across folders makes your standard hard to learn.
• No versioning. When you change a rubric, you need to know which version an agent used on which job.
• Weak visibility. If you can't see why an agent flagged something, you can't fix it. You can only restart it.

Fix these before you scale past the pilot. A system that works on ten cases with no logging falls apart at a hundred.

What to build it with

You don't need to build everything from scratch. You do need pieces that work together.

At minimum you need:

• An orchestration layer that sequences handovers, not standalone bots working alone.
• A review workspace that shows what is waiting, not a plain inbox.
• Audit logging built in from day one.
• Confidence thresholds you can adjust.

Our default build tool is Claude Code. A founder can turn a written process into Claude skills, chain those skills into agents, and put hard gates in front of anything risky. Claude Code hooks run before a tool call and can block it. So an agent can prepare a client email, but it can't send one without a person saying yes. We walk through that build path in custom AI delivery systems with Claude Code.

Off-the-shelf automation tools still have a place. They handle one slice well. But they leave the orchestration and governance to you, and that gap is where most pilots stall. The Devwiz team's breakdown of when to put a person in front of AI is a good companion read on that plumbing. New to terms like "orchestration layer"? Our glossary explains them plainly.

Our take: judgement is the asset worth protecting

We run our own business this way. The agents in our stack draft. A person pulls the trigger on anything that gets sent, posted or paid. That isn't caution for its own sake. It is what lets the agents take on more each week without the quality sliding.

The way I work with AI is 10/80/10. I do the first 10%: the plan, the context, the examples. The AI does the 80%. Then I do the last 10%, checking it and making it mine. Most people skip one of the two. Skip the first and you treat AI like a magic wand. Skip the last and you get slop.

A human in the loop system is 10/80/10 built into your operations. The first 10% is your codified rubric. The last 10% is your review gate. I say it to my own team all the time. Human in the loop is extremely important. We don't set and forget, and nothing ships until it has been tested.

The mistake I see most isn't technical. It is founders using AI as a shortcut past their own judgement, instead of a way to multiply it. Encode that judgement into an AI Operating System of AI employees and your standard scales without you in every loop. The Njin team's piece on what an AI operating system for business is covers the people and process side of that idea.

James Killick

Get a 90-day plan built around your standard

Advice from the sidelines won't give you a working system. Neither will a generic AI tool you hope learns your style.

Our program for coaches and consultants is a 90-day, done-with-you build. We run discovery, codify your rubrics, prototype the agents, and hand over a system your team can run. If you want the evidence first, start with our research.

The first step takes six minutes. Take the Founder Bottleneck Assessment to score how much of your business still runs through you, and get your next move.

Sources

• Building effective agents (Anthropic)
• Article 14: Human oversight (EU AI Act)
• Hooks reference (Claude Code docs)
• Human-in-the-loop patterns (Zapier)
• AI tools deliver more when you design for humans in the loop (Slalom)
• Professional services multi-agent AI implementation guide (SysGenPro)
• Field Orchestrator launch (Fieldguide)