LLM Selection: Test 20 to 50 Real Queries Before You Pick

Pick the smallest model that reliably does the job inside your cost and governance limits. That is the whole answer. The rest of this post shows you how to prove it.

Here is the short version. Build a small test set. Run 20 to 50 of your real queries through a shortlist. Score each model on three things: does it do the work, can you afford it, and is it safe to run on your data.

A fast LLM selection pipeline for busy founders

The pipeline has six steps. Define the task. Pick a metric. Shortlist models. Test them. Choose one. Keep checking it.

Each step feeds the next. Skip one and you pay later, usually with a model that looked great on paper and broke in production.

Before any testing, check four hard limits. Think of them as a locked door. No key, no point picking the lock.

• Complexity: how hard is the task? A simple sorter does not need a giant model.
• Skills: can your team monitor and fix the model you pick?
• Budget: can you afford the token cost at real volume, not demo volume?
• Governance: does the model meet your data, privacy and audit rules?

These four checks come from the Microsoft AI Decision Framework, which calls them hard constraints. Get them wrong and no leaderboard will save you.

Public benchmarks have one job. They cut a long list into a short one. They can't tell you if a model works on your task, with your data. Your own test does that.

Turn your task into numbers you can test

You can't pick a model until you know what good looks like. So write it down first.

Start with one primary metric. It is the single number that says the model did its job. For a summariser, it's factual accuracy. For a sorter, it's precision and recall. For code, it's whether the code runs and passes its tests.

Then add secondary metrics. They catch what the main number misses:

• Format: does the output match the structure you need?
• Latency: how long until the answer lands?
• Cost per call: what does each request cost at your volume?
• Consistency: do similar questions get similar answers?

Put it in a simple table before you test anything:

Task	Primary metric	Pass rule	Cost ceiling
Support ticket sorter	Accuracy	95% correct on 40 tickets	Set per call
Contract summary	Factual correctness	Zero invented clauses	Set per call
Code helper	Tests pass	Passes on first run	Set per call

Write the pass rule before you run a single test. It stops you moving the goalposts when your favourite model slips.

Use public benchmarks to screen, not to decide

Think of a benchmark as a CV. It tells you who is worth an interview. It doesn't tell you who can do your job.

HELM (Holistic Evaluation of Language Models) is a good screening tool. It doesn't rank on one score. It measures accuracy, calibration, robustness, fairness, bias, toxicity and efficiency together, so the trade-offs show up.

That matters. A model can top an accuracy chart and still be slow, pricey or confidently wrong. Calibration is the word for knowing when you don't know.

Use benchmarks for three jobs only:

• Cut a long list down to five or six worth testing.
• Spot known weak areas, like reasoning, code or safety.
• Check a model's general class before you spend time on it.

Then stop. The final call needs your own data.

A simple screening rule works well. Drop any model that sits in the bottom half on your key skill and also busts your budget. That alone cuts most lists in half.

Score models on more than accuracy

Accuracy is one axis. You need a few more.

Score each shortlisted model on these:

• Task utility: does it solve the problem you have?
• Calibration: does it flag doubt, or guess with confidence?
• Robustness: does it hold up on messy, real inputs?
• Latency and cost: is it fast enough, and cheap enough at volume?
• Privacy: where does your data go, and who can see it?
• Hallucination risk: how often does it invent facts, and how much does that hurt?
• Oversight: how much human checking does it need?

Weight the axes by risk:

• Internal tool (a meeting summariser): weight cost and speed.
• Customer-facing tool (a support bot): weight hallucination risk and robustness.
• Regulated work (health, finance, legal): weight privacy, calibration and oversight. Treat a weak score there as a hard fail.

Score each axis 1 to 5, multiply by its weight, and add it up. Set a minimum total. Anything under it stays out of production, whatever the leaderboard says.

Treat privacy and governance as pass or fail. Never let a high accuracy score average them away.

Build a short test and pick with a scorecard

You don't need thousands of examples. You need the right handful.

Here is the process:

1. Screen with benchmarks to get a list of 15 or 20.
2. Apply hard filters (budget, data location, speed) to get 6 to 10.
3. Pick three to six finalists using your rubric.
4. Write 20 to 50 test queries that cover edge cases, not just easy wins.
5. Run every finalist and score it on the scorecard.
6. Pick the model that clears every hard limit and scores best on utility.

Step 4 is where teams burn money. Hundreds of random queries is slow and dear. Pick queries that tell you the most instead. The best ones are queries where models disagree. If every model gets it right, the query teaches you nothing.

Your scorecard should hold:

• Model name and version
• Pass rate on the primary metric
• Pass rate on secondary metrics
• Weighted rubric score
• Hard limit status (pass or fail, no partial credit)
• Notes on edge case failures

Running more than one model in one system? You don't need your best model everywhere. Give cheaper models the low-stakes steps, and save the top model for the step that needs judgement. Anthropic's guide to building effective agents makes the same point: start simple and add complexity only when it earns its place.

Need a test rig to run these tests again and again? Our post on AI agent testing scenarios and CI checks covers the replay patterns. Anthropic's docs on defining success criteria and building evaluations are a solid second reference.

Why LLM selection belongs inside an AI Operating System

Here's the thing most guides miss. You're not picking one model for one chat window. You're picking the brains behind a team of AI employees.

In an AI Operating System, each AI employee has a job: draft the proposal, sort the inbox, check the numbers. Each job needs a different level of model. A routine sorter can run on a small, cheap model. A step that carries your judgement, the bit your clients pay you for, earns the stronger one.

That's why we build these systems with Claude Code. It lets you write the test queries, run them across models and keep the scorecard in your own project, without a developer. When a vendor ships a new version, you rerun the same tests and swap the model in one place. Our post on building custom AI delivery systems with Claude Code shows what that looks like. Njin's guide on what an AI operating system for business is is a good plain-English primer too.

Check load, quotas and real running cost

A model that works in a demo can fall over in production. The model is rarely the cause.

The trouble is the plumbing: knowledge sources, connectors, tool calls and rate limits. The Microsoft framework flags that quotas, tool latency and throttling decide whether a solution fits, so test the full path and not just the model's raw speed.

Your load test should cover:

• Peak load: what happens at your busiest hour?
• Retrieval speed: how fast does your knowledge base answer under load?
• Tool chains: if the model calls three tools in a row, what's the total time?
• Rate limits: what happens when you hit a quota mid-task?

Then work out the real bill. Inference is only part of it. Add input and output tokens at your true volume, retries, storage for logs and history, and monitoring tools.

You can bring it down. Cache repeat questions, send simple jobs to a smaller model, and trim your prompts. Our guide to cutting LLM inference costs goes through each one.

Also check data rules before you sign. If your data must stay in one region, or can't train a vendor's future models, get that in writing.

Keep governance running after launch

Picking a model isn't a one-off job. It's the start of a watch.

NIST's Generative AI profile sets out what good looks like. It points to documented data origins, capability testing, ongoing evaluation, privacy and IP controls, and clear rules for when a human steps in.

Build these in from day one:

• Data lineage: know where every input came from.
• Human review rule: which outputs need a person to check them.
• Incident plan: what happens when the model gets something badly wrong.
• Scheduled re-tests: not just one test before launch.

Set triggers for a fresh round of testing. A new model version, a drop in output quality, or any incident report should send you back to your original scorecard.

Keep simple logs too: sample input and output pairs, flagged failures, speed over time, and cost per call. They let you catch drift before a customer does.

Put a reminder in your calendar to re-test every quarter, even when nothing has gone wrong. Model behaviour shifts quietly. For a deeper playbook, see our 90-day governance playbook for CISOs.

Where teams go wrong

The same mistakes show up again and again:

• Chasing the top benchmark score instead of testing real tasks.
• Leaving the governance check until the model is live.
• Testing on easy queries that every model passes.
• Ignoring running cost until the first invoice.
• Picking the biggest model "to be safe", then paying for power nobody uses.

Pin one rule above your desk. Pick the smallest model that clears your thresholds. Not the most capable one.

One last warning. Lock-in rarely shows up in testing. It shows up six months later, when prices change or a model is retired. Build so that switching models is a settings change, not a rebuild.

Ready to run this on your own business?

Reading a selection pipeline is one thing. Running it on your own workflows is another.

That's what the Foundation Month and 90-day program are for. We work with $1M+ educators and consultants to turn their know-how into an AI Operating System of AI employees, tested on real tasks and built with Claude Code. You keep the tests, the settings and the playbooks.

Want a second pair of eyes on your model choice? Book an AI readiness assessment and we'll look at your task together.

Sources

• Holistic Evaluation of Language Models (HELM)
• Artificial Intelligence Risk Management Framework: Generative AI profile (NIST)
• Evaluation Criteria | Microsoft AI Decision Framework
• Building Effective AI Agents (Anthropic)
• Define success criteria and build evaluations (Claude Platform Docs)