How to Compare AI Models with a Repeatable Test

The short answer

A useful model comparison uses tasks the evaluator understands, a scoring rule chosen in advance, identical inputs where possible, and records that another person can inspect. A list of vendor benchmark wins is not a comparison. Neither is one impressive screenshot.

This guide sets out the method we use for model comparisons on AI Tool Herald. It is written so any team can apply it to its own work.

Why it matters

AI results vary with the prompt, tool access, model version, temperature, context, and surrounding application. A screenshot of one answer captures almost none of that.

Google’s documentation on its reviews system describes the kind of review it aims to reward: content that “provides insightful analysis and original research” and is written by experts or enthusiasts who know the topic well. It says the system works to show in-depth research rather than thin content that simply summarizes a bunch of products or services. It also names head-to-head comparisons and ranked lists as review formats. That standard is good for readers whether or not search traffic is the goal.

Step 1: Write the decision question

Start with the decision the reader faces. “Which model is best?” cannot be tested. “Which model should a five-person support team use to draft replies from its help center, within a fixed monthly budget?” can.

The question sets the audience, the task type, the constraints, and the level of risk. Everything else follows from it.

Step 2: Build the task set

Choose ten to twenty tasks from the audience’s real work. A balanced set includes:

• Routine tasks that represent most of the daily volume.
• Hard cases with long inputs, conflicting sources, or several steps of reasoning.
• Ambiguous instructions, to see whether the model asks or guesses.
• A missing-information task, where the correct response is to ask for what is missing rather than invent it.
• A boundary case, where the model should decline, warn, or flag a risk.

Keep the inputs fixed. Save them as files so the same text goes to every model.

Step 3: Define success before running anything

Scoring rules written after seeing the outputs tend to favor whichever answer looked best. Decide in advance.

• Factual work: list acceptable sources and the facts a correct answer must contain.
• Code: provide tests the output must pass.
• Writing: list required facts, the required tone, and prohibited claims.
• Extraction: provide a ground-truth answer set and score field by field.

Use a simple scale, such as 0 for a failure, 1 for usable with edits, and 2 for usable as is. Fine-grained scores imply precision the evidence rarely supports.

Step 4: Record the environment

Save the model identifier, test date, product plan, system instructions, prompt, attached files, enabled tools, and relevant settings such as temperature. Note whether you used an API or a consumer app, because the same model name can behave differently in each.

If a provider updates a model without changing its name, record that limitation. Run tasks in a fresh context unless memory is part of the feature being tested. Use more than one run per task when variability matters, and report the spread, not only the best result.

Step 5: Score the whole workflow

The output is only part of the cost. Measure each of these:

Criterion	What to record
Correctness	Score against the predefined answer or tests
Completeness	Required elements present or missing
Unsupported claims	Statements not backed by the inputs or sources
Instruction following	Format, length, and constraint violations
Latency	Time to a usable answer, including retries
Direct cost	API or plan cost for the task
Review time	Minutes a person spent checking and correcting

Total cost includes review and correction time, not only the API price. A cheaper model that needs ten minutes of fixing per answer can cost more than an expensive one that needs two.

Weight the criteria according to the audience. A fast model with a higher factual error rate may be unacceptable for research and excellent for low-risk formatting.

Step 6: Blind the scoring where you can

Remove model names from outputs before scoring, and shuffle the order. A reviewer who knows which model wrote an answer may read it through prior expectations. When two people score, compare their results and discuss disagreements before finalizing.

Step 7: Publish evidence, not theater

A published comparison should show:

1. The decision question and the audience.
2. The test date and the exact model identifiers.
3. Representative prompts and outputs, including failures.
4. The scoring rule and the weights.
5. Known limitations, such as a small sample or a single run.
6. Any affiliate relationship or vendor-supplied access, disclosed near the recommendation.

Avoid naming a universal winner when results depend on the task. “Model A for long-document extraction, Model B for short drafting” is often the honest answer. For disclosure wording and placement, see our guide to affiliate disclosures for AI tool reviews.

Step 8: Re-run on a schedule

Models change. Date every result and re-run the task set when a tested model changes materially, or note on the page that the result may be out of date. Fixed inputs and fixed scoring rules make the re-run quick and the change easy to see.

Common mistakes

• Comparing one model in a chat app with another through the API.
• Tuning the prompt for one model and reusing it unchanged for the others without saying so.
• Reporting vendor benchmark numbers as if they were your own results.
• Treating one run as a result for a task with high variability.
• Leaving out human review time.
• Changing the scoring rule halfway through because a favorite model is losing.

The conclusion

The best comparison helps a reader reproduce the decision. It states the question, fixes the inputs, sets the scoring rule in advance, and shows the evidence. It does not ask the reader to trust a score whose method remains hidden. Our AI tool review methodology applies the same principles to full product reviews.