Methodology
One pinned reference agent runs every task against every provider identically (bounded loop, max 8 search calls per task); LLM judges grade the answers and are themselves audited against human labels before anything publishes. Nothing is blended into one number: the Agentic Search Index is a pure quality construct, and cost and latency are separate published axes.
Judge Calibration
Two independent humans grading the same answers agreed 95.8% of the time (Cohen's kappa 0.917, n=48). That is the ceiling any judge can reach. Our judge matched the human consensus 94.1% of the time (kappa 0.85, n=34). A second judge model agreed with the pinned judge's pass-fail call 95.1% of the time across 122 overlapping cases. 3,267 runs were executed and 3,263 verdicts were scored after voiding harness-fault runs.
| Comparison | n | Agreement | Kappa |
|---|---|---|---|
| Human vs human (ceiling) | 48 | 95.8% | 0.917 |
| Judge vs human consensus | 34 | 94.1% | 0.85 |
| Judge vs second judge model | 122 | 95.1% pass-fail | n/a |
Task Bank
| Kind | Tasks | What it tests |
|---|---|---|
| Static Facts | 45 | Short factual questions with one indisputable answer. |
| Dynamic Facts | 28 | Questions whose answers change with the world. |
| Multi-Source | 40 | Questions requiring integration of multiple sources. |
| Deep Research | 8 | Hard multi-constraint discovery questions. |
The scored split is private and rotates per release; the freshness generator is private, so providers cannot train on the test. A task passes only if it addresses the question, is correct, is current where that applies, and is supported by a returned source. Fabricated support scores minus 1: fabricated confidence is worse than an honest miss. Provider errors and timeouts count against the provider; harness faults on our side are voided and re-run, never scored.
Verdict Buckets
| Verdict | Meaning |
|---|---|
| Correct Answer | Answered the task correctly. |
| Question Not Addressed | Never addressed the question asked. |
| Incorrect Answer | Answered confidently, but incorrectly. |
| Outdated Answer | Answered with out-of-date information. |
| Fabricated Citation | Cited a source that does not support the claim. |
| Provider API Error | The provider call itself failed. |
Baselines
A no-search baseline (the identical pinned agent with search disabled) scored 16.7 on the same bank.
Cost Method
Cost per 1,000 successful answers = billed provider fees across the whole workload (all calls, including retries and failed tasks) plus payload ingestion priced at the pinned shopper's input rate (Claude Opus 4.8, $5.00 per million tokens), divided by successful tasks.
Worked example (Firecrawl): $0.01 list price/call x 2.4 avg searches per task / 0.83 pass rate = $0.0379 per successful answer ($37.94 per 1,000).
Versioning
- Kind weights and the judge (model and prompt) are pinned per index version; any change ships as a new, recalibrated version.
- Task splits are private and rotate per release; saturated sources are retired with published rationale.
- Changes and corrections are logged in the changelog, never silent.
Telemetry Semantics
- Liveness is not "returns 200": every watched surface carries an expected status set, and a 200 with the wrong content records as a soft-404, not presence.
- Blocked is not down: bot walls are recorded as blocked and never counted as downtime.
- Percentiles come from rollups only; a single-sample latency is never displayed. Speed and liveness are measured with unpaid probes from a single vantage point (our probe region). Treat them as our view, not global ground truth.
- The watcher never pays and never authenticates: unpaid, unauthenticated requests only.
Lineage
The judge design adapts the Universal Verifier's published calibration principles (Rosset et al., arXiv:2604.06240); hurdle and signed grounding follow Mercor's ACE. Task-bank seeds: SimpleQA-Verified (MIT), FRAMES and WebWalkerQA (Apache-2.0).