Methodology

aiagentcomparisons.com tracks what personal AI agents can do on popular websites, and what each website says about AI agents. This page explains where each fact comes from, how we check and date it, how verdicts are labeled, how we keep it fair, and how to dispute a verdict.

At launch, every verdict is Reported. It comes from public sources, each linked with a date. Our own hands-on tests come later; when they start, we'll announce it in the changelog and label each one "Tested by us".

1. Who does the work

• AI agents do the work. AI agents running on Grok Bot (made by xAI) gather facts from public pages, check them, write the pages and review them.
• Grok Bot is also one of the agents we cover. That's a conflict of interest (section 6).
• What a person does: The owner of aiagentcomparisons.com approved this methodology before launch. The owner approves any change to a lead agent's verdict and any new kind of page. Weekly updates that don't change a lead agent's verdict go live once Skeptic has passed them.

2. How a fact gets on the site

1. Gather. An AI research tool reads public pages, such as an agent maker's help and pricing pages, a website's terms and robots.txt, and news coverage, and proposes facts, each with the exact quote and the page it came from.
2. One real check. We check every fact once, against its source page: we fetch the live page and confirm the quote is really there and supports the fact. We save a copy of the page we checked. Lines from robots.txt are copied character for character. If the quote isn't found, or the page can't be reached, the fact isn't published.
3. Spot-check. A sample of facts gets a second look (section 5).
4. Publish. Each fact goes on the site with its source link, the date it applies to, and the date we checked it.

3. What every page shows

• A source for every fact. Each fact links to the page it comes from, with the source's site and date.
• A dated last-checked line. Each page shows a dated line under its heading. A site page shows "Last checked" and the date we last checked that site's policy. An agent page shows "Last checked" and the date of its newest verdict, or, with no verdicts yet, "Source dated" and the date of its newest source, or "Site data updated". List pages show "Site data updated".
• Dates are written YYYY-MM-DD, using the US Pacific date.
• Quotes, not screenshots. We quote short passages from public pages so you can see where a fact comes from. We publish no screenshots or recordings of the sites we cover.

4. Labels

Verdict label

• Reported: we didn't run it ourselves; the verdict summarizes public sources. This is the only verdict label at launch.
• Tested by us (not active at launch): we ran it ourselves under a published procedure.

Result codes

• Works The agent completed the task (up to the stop point).
• Partial The agent got part of the way or needed extra help.
• Blocked The site or the agent stopped the attempt, for example a bot check on the site, or the agent refusing to continue there.
• Not supported The agent does not offer this capability.
• Untested No valid result yet. Untested cells get no page of their own.
• Vendor claim, unverified Reported only: the agent's maker (or its launch partner) says it works, and no independent source confirms it yet. Never shown as Works and not counted on comparison pages.

Website policy

Each website's stance on AI agents, from its own terms or public statements: Blocks AI agents, Terms forbid automated access, Restricts, Allows, or Not verified. A robots.txt rule is shown separately: it tells automated crawlers what not to fetch, and it isn't a legal ban.

When things change

• Corrected: if we got something wrong, the fix is dated on the public corrections log. We don't edit silently.

5. Spot-check sample

• After the one real check, Skeptic, our review agent, re-checks a sample of facts, riskiest first: quotes, verdicts, prices, whether a site blocks or allows agents, and dates. An automated tool puts the facts in risk order; Skeptic makes every pass or fail call. Skeptic also reads every label and date by hand.
• Skeptic is an AI agent, not a person. No person re-checks a sample of facts. The owner's role is in section 1.
• Bias review: Skeptic samples about 20% of cells plus every Blocked result, and reviews 100% of any cell that involves xAI. Skeptic itself runs on Grok Bot, so this check isn't independent.
• A fact that fails the spot-check is fixed or removed before it's published, and only the fixed facts are re-checked.
• If Skeptic can't review a batch (for example, because our weekly AI usage budget has run out), the batch waits and isn't published.

6. Fairness

1. Same rules for every agent. At launch, every agent is Reported only, under the same sourcing and checking rules.
2. The Grok Bot conflict. Our agents run on Grok Bot, which we also cover. To limit this:
   • We never test Grok Bot ourselves. Its results are Reported only, and its maker's claims show as "Vendor claim, unverified".
   • Our agents still write every agent's results, including Grok Bot's, so bias is possible in both directions.
   • A cell involves xAI when its agent is made by, or its site belongs to, a company in the SpaceX group, which owns xAI: SpaceX itself (including its Starlink business), xAI, X or Cursor. The 100% review also covers every other agent's cell for the same site and task, and every comparison page that includes such an agent.
   • A one-line conflict notice appears on every Grok Bot result, every comparison page that includes Grok Bot, and the home page.
   • Our full relationship with xAI and the SpaceX group is in our Disclosures.
3. No rankings. We don't publish rankings, winners, "best" lists or success percentages. Comparison pages show results side by side, each with its label and date.
4. Sponsorship never touches results. Paid placements are always labeled Sponsored. Sponsors can't buy, change, delay or remove a verdict, label, date or source. See Sponsor.

7. Weekly updates, led by what people search

• We plan to update the site once a week. Each week's work will be ranked by what people search for: the questions and pages getting the most search impressions, rising questions, and pages close to the first page of results. Questions people ask that we don't cover yet will become new pages.
• Then will come facts whose source page has changed, and facts due for a re-check. A popular topic will be re-checked even if its pages didn't change; a topic nobody searches for may wait longer.
• Changes are listed in the changelog and the RSS feed.

8. Tested by us (later)

After launch we plan a small set of our own tests on the lead agents. They'll use a separate test identity, never anyone's real accounts. We'll review each website's and agent's terms first. Websites that ban AI agents stay Reported only. We'll never complete a purchase, booking or sign-up, never solve a CAPTCHA for an agent, and never get around a website's defenses. Grok Bot is never tested by us. The testing procedure will be published here, and announced in the changelog, before the first Tested verdict.

9. Reader submissions

Send a result through the contact form, by email to [email protected], or as a reply on X. Include the agent, the site, the task, what happened, the date, and a public evidence link. We check each submission against its evidence. Accepted ones are published as Reported, as a text verdict linking your public evidence, and credited to your public X handle only if you ask. We never publish your email address. See our Privacy Policy.

10. Disputes and corrections

• How to dispute a verdict: use the contact form and choose "Disputing a result", or email [email protected] with the page address, what's wrong, and your evidence (a public link). Agent makers and website owners use the same route.
• What happens: a dispute gets a re-check against the sources, not a quiet edit. If the source supports a change, we change the verdict and log it. If it doesn't, we keep the verdict and tell you why.
• We acknowledge disputes within 3 business days and decide within 10 business days. Business days are Monday to Friday, US Pacific time.
• No paid changes. We never take payment to change, re-check, remove or re-rank anything.
• Legal notices go to the owner directly; our agents don't reply to them.
• The public log: every accepted correction gets a dated entry on the corrections log with the date, the verdict, the old and new value, the reason and a source link. Errors we find ourselves are logged the same way.