Methodology

Two kinds of listing — and how to tell them apart

Every server on this site is either Listed or Tested. Listed means exactly one thing: the entry comes from the official MCP Registry (currently 16484 servers), with its description, origin, and transport data shown as published there. We make no quality claims about Listed servers.

A Tested badge appears only when a server has been run through our benchmark harness and the results were explicitly promoted for publication. This rule is enforced in the site's build: pages are physically unable to render measured numbers or testing claims without a promoted result artifact behind them.

How a benchmark run works

Servers are benchmarked per category, because comparisons are only meaningful between services that do the same job. Each category has a task suite in which every service receives byte-identical task inputs; per-service adapters translate those inputs to each server's tool calls without altering their meaning. Timeouts, retry rules, and concurrency are identical for every service. A capability a service does not offer is scored N/A and excluded from its denominator — never counted as a failure.

Scoring is deterministic first: task success, output validity, latency (p50/p95, measured on the tool call only), and metered cost. A separate, clearly-labeled advisory quality score from an LLM judge may accompany it, and is never blended into the success rate.

Keeping results honest

• Drift quarantine. Tasks that target live web content are validated by canary checks before any service is scored against them. If the target changed — or every service fails the same task — that task is quarantined for all services rather than counted against any of them.
• Publish gate. A run whose canary health falls below threshold cannot be promoted to the site at all.
• Reproducibility. Each published run pins its harness version, task-suite version, and the exact server package versions measured, and results pages display the run identifier.

Published benchmark data currently covers 1 category.