THE EVIDENCE BEHIND THE CHOICE.

Know what a number means.

A useful comparison makes its assumptions and its limits visible.

In 60 seconds
  1. 01Five separate outputsEach is reported on its own and never collapsed into a single number.
  2. 02Price is evidenceIt is recorded, never a ranking input.
  3. 03Rank like with likeOnly inside a cleared peer group; stubs and provisional boards are not ranked.
  4. 04Missing stays missingUnscored does not mean poor quality; unknown prices do not mean free.
  5. 05Decisions are edition-pinnedA missing or unverifiable edition fails visibly.

CITE THIS METHOD

Method AIS-V2

Identifier
ais-v2
Edition
2026.09-v2
Status
phase 1 live
As of
Sep 6, 2026

Rank like with like. Keep quality, capability, user outcomes, and adoption separate.

Price is recorded as evidence and is never a ranking input. Peer boards stay provisional or building-cohort until their declared evidence baskets clear. Building-cohort stubs are not scored and have no /best-* guides.

Five separate outputs

Each output is reported on its own and never collapsed into a single number. Price is not one of them.

  • Lab performancemay rank inside a cleared peer grouplab
  • External benchmark consensusmay rank inside a cleared peer groupexternal
  • Verified capability assessmentmay rank inside a cleared peer groupcapability
  • Qualified user outcomesmay rank inside a cleared peer groupusers
  • Adoption and momentummay rank inside a cleared peer groupadoption

Peer boards

The original five peer groups remain the live Method boards. Five additional boards are building-cohort stubs with empty membership.

Peer groups in this edition

Visual app builders provisional

Go from prompt or visual input to a hosted working app

7 members

Required evidence basketDesign Arena BuilderAI Scoreboard responsive build packLighthouse accessibilityRegression suite

Design to code provisional

Turn a design artifact into maintainable repository code

5 members

Required evidence basketVisual fidelitySemantic structureResponsive behaviorComponent reuseRepository integration

AI IDEs and copilots provisional

Work interactively inside an editor or existing repository

10 members

Required evidence basketSWE-bench ProTerminal-Bench 4.0Held-out repository tasksRegression rateIntervention rate

Autonomous coding agents provisional

Delegate implementation, testing and pull-request work

7 members

Required evidence basketSWE-bench ProTerminal-Bench 4.0Long-horizon repository tasksCostHuman interventions

General agent platforms building cohort

Run persistent work across desktop, terminal, browser, tools and schedules

1 member

Required evidence basketToolathlon-VerifiedMCPMarkOSWorld 2.0WebArena-VerifiedTerminal-Bench 4.0Agents' Last Exam

Building-cohort stubs5 boards, no members

Building-cohort stubs

These boards exist so the job and evidence basket are public before anyone is admitted. members[] stays empty until job-fit and evidence baskets clear. Catalogue intake candidates are not Method members.

Browser & computer-use building cohort

Operate a real browser or desktop GUI to complete multi-step tasks on software that has no clean API

0 members. Candidates are not membership. No quality rank, score, Elo, or /best-* guide is published until this board is provisional and its evidence basket clears.

Required evidence basketOSWorld 2.0WebArena-VerifiedBrowserGymWebVoyagerAgentic web task suites with fixed seeds

Voice agents building cohort

Run real-time speech agents (Live/Realtime APIs) that converse and take actions — not TTS/voiceover alone

0 members. Candidates are not membership. No quality rank, score, Elo, or /best-* guide is published until this board is provisional and its evidence basket clears.

Required evidence basketLive/Realtime task suitesLatency + barge-in robustnessTool-calling success in voice sessionsVendor Live API model cards with dated rates

Evals & observability building cohort

Trace, evaluate, and monitor LLM/agent quality, cost, and regressions in development and production

0 members. Candidates are not membership. No quality rank, score, Elo, or /best-* guide is published until this board is provisional and its evidence basket clears.

Required evidence basketDataset/eval runner docsProduction tracing docsOpen-source vs hosted scopeDated pricing or license pages

Agent memory & context building cohort

Persist, retrieve, and manage long-horizon agent memory and context beyond a single session window

0 members. Candidates are not membership. No quality rank, score, Elo, or /best-* guide is published until this board is provisional and its evidence basket clears.

Required evidence basketMemory API/docsRetention/privacy scopeBenchmark or case studies with dated sources

MCP & tool servers building cohort

Expose tools and data to agents via Model Context Protocol (or equivalent first-class tool-server products)

0 members. Candidates are not membership. No quality rank, score, Elo, or /best-* guide is published until this board is provisional and its evidence basket clears.

Required evidence basketMCPMarkToolathlon-VerifiedOfficial MCP server list / registry entriesAuth and scope documentation

Scope notes

  • Anthropic Computer Use and OpenAI Computer Use / CUA are capability and documentation identities, not standalone SaaS products, so they are not Method members.
  • Google Antigravity stays on AI IDEs and copilots pending dual-job review and is not auto-cross-listed onto browser & computer-use.
  • Voice agents exclude TTS-only products. OpenRouter is routing metadata, not a vendor price.
  • This page restates the published method. It does not invent scores for unscored models or convert a building-cohort or provisional board into a rank.

Kinds of evidence

Every figure belongs to one of four kinds, and the kind decides what it can support.

01

Editorial assessment

Saved judgments about task fit. Scores depend on the selected lens and available assessment dimensions. They are not newly measured benchmark results.

02

Source and vendor reports

Facts captured from linked documentation, product pages, and published reports. A check date records the observation—not a guarantee that the underlying claim is independently proven.

03

Measured observations

Results must retain their model version, provider, settings, dates, and test conditions. Different harnesses and workloads are not silently treated as equivalent.

04

Workload estimates

Calculated from documented token rates and your assumptions. Estimates are not invoices and do not include undocumented discounts or unpriced services.

Missing information stays missing

Unscored does not mean poor quality. Unknown prices do not mean free. Hard requirements exclude candidates when the required fact is not documented.

What the price calculation includes

Input tokens × the applicable input rate, with a documented cache-read discount where available, plus output tokens × the output rate. Per-workload cost is multiplied by the number of monthly workloads. Per-request prompt size selects applicable context tiers.

Usage totals may span many requests. Cache creation, cache storage, tools, media, taxes, and service-specific overhead are excluded. When a cache-read rate is not documented, input uses the full listed rate. Saved prices are included only by explicit choice.

Repeatable, edition-pinned decisions

Shared comparisons include the catalogue edition and workload assumptions. Opening a missing or unverifiable edition fails visibly; a smaller or different catalogue is not substituted.

Current catalogue edition
Edition
next-55a6b6f73329b9b2b98eed2ba6c28d3be0db02c5259e9918dae31a931742d5ab
Source release
studio-80091062dd0e86f942323457298c969b0dc845ccc27a2d64a9647f2a5ae4be06
Source commit
4fd649e62d63e33d8c44fe5a55c8ccfaa09b1193

Privacy and storage

Your shortlist and notes are local by default. Backups include private research. Cloud snapshots are explicitly requested and encrypted in the browser; their full links are credentials. Test prompts and submitted requests are sent to the existing service only on submission.

Commercial policy: money never buys rank

Submit or claim a listing is free. It waits in the editorial queue and does not auto-publish.

Report a correction

Corrections filed

1 records

Anyone can report a catalogue error. Each accepted report gets a stable CORR reference, a scope, and a visible disposition (open, fixed, not-a-defect). Fixed entries link the superseding record or release. Method changes create a new series; old snapshots stay accessible and labelled. This log is operational and loads fail-open: if it is missing, the catalogue still renders and the dialog says so.

Current corrections log; distinct from the edition-pinned catalogue. Triage owner: Nick (site owner).

CORR-2026-001 · Duplicate evidence records in audit coveragefixed
disposition
fixed
filed At
2026-09-12
fixed In
Current catalogue edition: the auditor dedupes evidence records by record id before counting coverage.
id
CORR-2026-001
issue
The catalogue auditor counted duplicate evidence records once per appearance, overstating canonical-record coverage.
scope
method
subject
Duplicate evidence records in audit coverage