The AI Tool Review Methodology We Will Use

The short answer

We will review a specific version or product state on a named date, using tasks selected for the intended audience. We will preserve prompts and key outputs, verify factual claims, and include limitations that could change the recommendation. This page explains each step, so readers can hold our reviews to the standard we set.

Why it matters

AI tools change quickly and often share similar marketing claims. A review is useful only when readers can see what was tested, what was not tested, and why the conclusion follows.

Google’s documentation on its reviews system points the same way. It says the system aims to reward reviews that provide “insightful analysis and original research” and are written by experts or enthusiasts who know the topic well. It works to show in-depth research rather than thin content that simply summarizes a bunch of products. It covers single-item reviews, head-to-head comparisons, and ranked lists. We would choose this standard even if search traffic were not a factor, because it is what makes a review worth reading.

Scope: what counts as a review

This methodology applies to future hands-on reviews on AI Tool Herald. Launch-news articles based only on official announcements are labeled as reporting or analysis, not independent product tests. If we have not used a tool, we will not write as if we have.

Step 1: Start with a reader question

Each review begins with a reader question: who should use this, for what job, and under what constraints? For example: “Is this meeting-notes tool accurate enough for a small consulting team that records client calls in English and French?” The question decides the tasks, the scoring weights, and the shape of the verdict.

Step 2: Design the test set

The test set includes three kinds of task:

• Ordinary work that represents what the target user does every day.
• A difficult edge case, such as a long document, poor audio, mixed languages, or conflicting instructions.
• A failure-recovery task, where we watch what happens when the tool gets something wrong. Can the user see the error, correct it, and undo any damage?

Where possible, inputs are fixed files that we can reuse when we retest. When a review compares underlying models, we will follow our repeatable AI model test.

Step 3: Record the setup

We record the plan, model, settings, enabled tools, input files, test dates, and whether the account was supplied by the vendor. We note the surface used, because the same product can behave differently on the web, desktop, mobile, and API.

We do not place confidential client data into a review account. Test inputs will be public documents, synthetic data, or material created for the test.

Step 4: Check privacy and data handling

Before any score is published, we read the tool’s current privacy policy and data settings and answer a fixed set of questions:

1. Is user content used to train models by default, and can that be turned off?
2. How long is content retained, and can users delete it?
3. Where is data processed or stored, if the provider says?
4. Are admin controls, audit logs, or single sign-on available, and on which plans?
5. Does the tool connect to third-party services, and what permissions does it request?

We report what the provider states and what we could confirm in the product. We do not present a provider’s policy as an independent audit.

Step 5: Score with visible weights

Scores may cover the criteria below. The weighting is explained in each article, because a solo writer and an enterprise legal team value these things differently.

Criterion	What we look at
Outcome quality	Accuracy and usefulness of results on the test set
Reliability	Consistency across repeated runs and edge cases
Usability	Setup time, clarity of controls, quality of error messages
Speed	Time to a usable result, including retries
Privacy controls	Answers to the data-handling questions above
Accessibility	Keyboard use, screen-reader support, captions where relevant
Total cost	Plan price, usage limits, and time spent correcting output

We avoid fake precision when the evidence does not support it. A score like 8.7 implies a level of measurement that a short test rarely has. We will prefer simple bands with a written explanation of what moved the score.

For models, total cost includes correction and review time. For subscription tools, the comparison uses the current public plan and notes important limits.

Step 6: Verify every factual claim

Feature lists, prices, limits, and supported languages are checked against the provider’s current documentation and, where possible, in the product itself. When the two disagree, we say so. Claims we cannot verify are removed or clearly marked as the vendor’s statement.

Commercial relationships

Affiliate links and sponsored access are disclosed close to the relevant recommendation. A vendor may correct a factual error before publication but does not receive approval over the conclusion. Affiliate availability does not decide inclusion, ranking, or the verdict. Our affiliate disclosure explains the policy in full.

Updates and corrections

Fast-changing reviews display a tested date and a last-reviewed date. A material product change triggers a retest or a visible note that the old conclusion may no longer apply. When a correction changes the conclusion, the article will say so, in line with our editorial policy.

What readers should expect

Each finished review will state:

• The reader question and who the tool fits.
• The version, plan, and test dates.
• The tasks, with representative prompts and outputs.
• What was not tested.
• Privacy findings.
• The scoring weights and the reasoning behind the score.
• Any commercial relationship.

Expect methods, evidence, and a scoped recommendation. Do not expect every new tool to receive a review, every feature claim to be repeated, or a universal winner chosen for conversion value.

The conclusion

A review earns trust by showing its work. We will name the version and date, test real tasks, check privacy before scoring, explain the weights, and disclose any commercial tie. When the product changes, the review will say so.