Skip to content
Eval Suites50

50 indexed suites • 15 technical guides • Primary sources linked

The independent registry of production AI eval suites

Compare 50 benchmarks, datasets, and harnesses for AI agents, RAG pipelines, code generation, and LLM security. Every listing links its official repo or paper, license status, stated dataset size, and runnable starter command; sponsored placements are labeled.

Harnesses and runners covered in our guides

  • Promptfoo
  • DeepEval
  • Inspect AI
  • Ragas
  • Pytest / GitHub Actions

Indexed suites

50

Registry

8 categories

OSI-approved licenses

33

Declared terms

66%

Harnesses & runners

5

Five tools

Indexed

Guides & comparisons

15

Editorial

4 clusters
Suite tier

50 of 50 suites

Fixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.

Size
500 tasks
Runs with
Docker / Inspect AI
License
MIT
Category
Agents

Multi-turn agents follow domain policy while talking to a simulated user. τ²-bench adds a telecom domain where agent and user both act on shared state (dual-control).

Size
165 tasks (τ) · 278 (τ²)
Runs with
Python
License
MIT
Category
Agents

GAIA

production

General-assistant questions that need web browsing, file reading, and multi-step tool use, with short exact answers.

Size
466 questions
Runs with
Inspect AI
License
Not stated
Category
Agents

OSWorld

frontier

Open-ended computer-use tasks on real desktop apps in Ubuntu, checked by execution-based scripts.

Size
369 tasks
Runs with
Sandbox VM
License
Apache-2.0
Category
Agents

Hard tasks in real terminal environments, from compiling code to configuring servers, run through the Harbor framework.

Size
89 tasks
Runs with
Harbor
License
Apache-2.0
Category
Agents

Scores simple, parallel, nested, and multi-turn function calls against expected calls and executable checks.

Size
Varies by version
Runs with
Python
License
Apache-2.0
Category
Agents

WebArena

frontier

Browser agents complete tasks on self-hosted e-commerce, forum, GitLab, and CMS sites, graded on final page or DB state.

Size
812 tasks
Runs with
Self-hosted sandbox
License
Apache-2.0
Category
Agents

WorkArena

production

Knowledge-work tasks inside ServiceNow: forms, lists, dashboards, and catalogs. WorkArena++ adds compositional tasks.

Size
33 task types (682 in ++)
Runs with
BrowserGym
License
Not stated
Category
Agents

An open RL-trained coding agent, not a benchmark. Its release reports results on SWE-bench Verified with test-based verifiers; listed here for its reproducible evaluation setup.

Size
Model, not a dataset
Runs with
R2E-Gym
License
Not stated
Category
Coding

LiveCodeBench

production

Competitive programming problems collected over time with release dates, so you can score only problems published after a model's cutoff.

Size
Grows by release
Runs with
Python
License
MIT
Category
Coding

KernelBench

frontier

Asks models to write GPU kernels for PyTorch workloads and checks both correctness and speedup over the reference.

Size
250 workloads
Runs with
Python / CUDA
License
MIT
Category
Coding

HumanEval's 164 problems with roughly 80x more tests, catching code that passed the original thin test suites.

Size
164 problems
Runs with
EvalPlus
License
Apache-2.0
Category
Coding

Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.

Evaluation suites by category

Choose the evaluation discipline matching your system architecture.

Registry mix by evaluation category

Each segment is a share of all 50 listings. Hover, focus, or tap for counts and examples.

Shares describe registry composition, not quality, popularity, or recent activity.

Declared license coverage

33 of 50 listings declare an MIT, Apache, or BSD license. The rest need a closer license check.

OSI-approved
33 · 66%
Other stated
3 · 6%
Not stated
14 · 28%
Other stated terms are CC-BY-NC-4.0, CC-BY-4.0, and CC0-1.0. Check each official repository before use.

How to choose an AI evaluation method

Production LLM evaluation can use three distinct testing layers: deterministic checks, model-graded judges, and end-state environments. Use the layer that can observe the failure you care about; an LLM judge should not be the only release gate.

Add execution scope only when the failure needs it

Start with direct checks. Move up a layer when semantic judgment or environment state is part of the behavior you need to test.

  1. 1. Deterministic assertionsCheck exact outputs and formats
  2. 2. Model-graded judgesAdd a judge model and rubric
  3. 3. Multi-turn end-state environmentsRun tools and inspect final state
The widths show increasing execution scope, not measured cost, latency, accuracy, or usage. Conditions and example tools remain in the table below.
Evaluation methods by use case, cost conditions, and example tools
Evaluation methodPrimary use caseCost & latencyBest suited tools
Deterministic assertionsExact format checks, JSON Schema validation, regular expressions, and explicit latency limits.[1]No model tokens for the assertion itself. Total runtime and cost depend on the application and runner; no universal sub-millisecond or zero-cost guarantee.Pytest, Promptfoo assertions
Model-graded judges (including G-Eval)Faithfulness, semantic tone, and rubric scoring where a deterministic answer is unavailable. Judge outputs need bias and agreement checks.[2][3]Provider token charges and judge latency vary by model, prompt, and configuration. Measure them on your own fixed test set.DeepEval, Ragas, Inspect AI
Multi-turn end-state environmentsAutonomous agents, tool execution, state changes, and multi-step API tasks. Inspect supports sandboxed execution; tau-bench compares final environment state with task goals.[4][5]Environment setup, model calls, and tool runtime drive cost. Docker or a VM may be appropriate, but neither is universally required; no universal minutes-per-run figure is stated.SWE-bench, tau-bench, Inspect Sandboxes

Featured evaluation guides & playbooks

In-depth technical guides with runnable configs, primary mathematical formulas, and CI/CD workflows.

Fundamentals & frameworks

Start with reference cases, a scoring method, and a runner you can maintain.

RAG & information retrieval

Separate retrieval quality, answer support, and metric implementation.

Agent & tool validation

Test state, tools, and repeated execution before release.

Security & vulnerability scanning

Probe untrusted inputs, disclosure, and agent permissions in authorized test environments.

Frequently asked questions about AI evaluation

Observability platforms such as Langfuse, Phoenix, and LangSmith primarily trace, log, and monitor application traffic. Evaluation suites run controlled test cases to measure behavior against references or checks, often before deployment. The categories overlap: some observability products also support evaluation, so compare the workflow you need rather than relying on the product label.[6][7][8]

Homepage sources

  1. 1. Promptfoo: Assertions and metrics · Promptfoo · Official docs · Sources checked
  2. 2. G-Eval metric · Confident AI · Official docs · Sources checked
  3. 3. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · arXiv:2306.05685 · Paper · Sources checked
  4. 4. Inspect AI: Sandboxing · UK AI Security Institute · Official docs · Sources checked
  5. 5. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · arXiv:2406.12045 (Sierra) · Paper · Sources checked
  6. 6. LLM observability · Langfuse · Official docs · Sources checked
  7. 7. Phoenix documentation · Arize · Official docs · Sources checked
  8. 8. Observability · LangChain · Official docs · Sources checked
  9. 9. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations · arXiv:2411.00640 (Anthropic) · Paper · Sources checked
  10. 10. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment · Yang Liu et al., arXiv:2303.16634 · Paper · Sources checked

Get new suites in your inbox

One email a week: new listings, benchmark changes, and one eval pattern you can use. Sponsored slots are labeled.