SWE-bench Verified
productionFixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.
- Size
- 500 tasks
- Runs with
- Docker / Inspect AI
- License
- MIT
- Category
- Agents
Compare 50 benchmarks, datasets, and harnesses for AI agents, RAG pipelines, code generation, and LLM security. Every listing links its official repo or paper, license status, stated dataset size, and runnable starter command; sponsored placements are labeled.
Harnesses and runners covered in our guides
Indexed suites
50
Registry
8 categoriesOSI-approved licenses
33
Declared terms
66%Harnesses & runners
5
Five tools
IndexedGuides & comparisons
15
Editorial
4 clusters50 of 50 suites
Fixes real GitHub issues in Python repos; a patch passes only if the repo's hidden tests pass. Verified is the 500-task human-validated subset.
Multi-turn agents follow domain policy while talking to a simulated user. τ²-bench adds a telecom domain where agent and user both act on shared state (dual-control).
General-assistant questions that need web browsing, file reading, and multi-step tool use, with short exact answers.
Open-ended computer-use tasks on real desktop apps in Ubuntu, checked by execution-based scripts.
Hard tasks in real terminal environments, from compiling code to configuring servers, run through the Harbor framework.
Scores simple, parallel, nested, and multi-turn function calls against expected calls and executable checks.
Browser agents complete tasks on self-hosted e-commerce, forum, GitLab, and CMS sites, graded on final page or DB state.
Knowledge-work tasks inside ServiceNow: forms, lists, dashboards, and catalogs. WorkArena++ adds compositional tasks.
An open RL-trained coding agent, not a benchmark. Its release reports results on SWE-bench Verified with test-based verifiers; listed here for its reproducible evaluation setup.
Competitive programming problems collected over time with release dates, so you can score only problems published after a model's cutoff.
Asks models to write GPU kernels for PyTorch workloads and checks both correctness and speedup over the reference.
HumanEval's 164 problems with roughly 80x more tests, catching code that passed the original thin test suites.
Size and license come from each suite's primary source. Not stated means the source doesn't declare a license; check the repo before commercial use.
Choose the evaluation discipline matching your system architecture.
Each segment is a share of all 50 listings. Hover, focus, or tap for counts and examples.
33 of 50 listings declare an MIT, Apache, or BSD license. The rest need a closer license check.
Multi-turn trajectory and environment-state testing, including tau-bench, SWE-bench, and GAIA.
Context retrieval, faithfulness, and answer-support metrics, including Ragas and FRAMES.
Direct and indirect prompt injection, canary leakage, and jailbreak tests, including AgentDojo and HarmBench.
Patch generation and test-driven execution benchmarks, including HumanEval and SWE-bench Verified.
Multi-step math, symbolic logic, and contamination-aware sets, including GSM8K and SimpleQA.
Finance, legal, and biomedical tasks with domain-specific references, including FinQA and PubMedQA.
Cross-modal grounding, audio tasks, latency, and strict formatting constraints, including IFEval and MMMU.
CLI tools and CI/CD test executors for custom suites, including Promptfoo and Inspect AI.
In-depth technical guides with runnable configs, primary mathematical formulas, and CI/CD workflows.
Start with reference cases, a scoring method, and a runner you can maintain.
Build a versioned LLM test dataset from real failures, reviewed synthetic cases and clear labels. Size it for uncertainty, not a magic minimum.
3 min read
Understand the original G-Eval paper, probability-weighted scores and a Python DeepEval example. Keep model availability and judge bias explicit.
2 min read
Use deterministic checks wherever the answer can be verified, LLM judges only for what can't, and pin both. How to cut flaky eval failures without losing coverage.
2 min read
Choose a testing tool by configuration, agent execution and scoring needs. Separate pre-release evaluation from production observability and measure costs.
2 min read
A plain comparison of three open-source eval harnesses: config style, sandboxing, metrics, red-teaming, and CI. Pick by your team's language and your riskiest failure.
3 min read
Separate retrieval quality, answer support, and metric implementation.
Separate retrieval relevance, answer support and question relevance. Compare Ragas faithfulness with TruLens groundedness without equating their scores.
3 min read
Compare DeepEval and Ragas for Python evaluation: CI integration, reference data, local versus network-free runs, licenses and judge costs.
2 min read
Test retrieval and generation separately, then add unanswerable questions on purpose. A checklist of RAG metrics, what each one catches, and where each one lies to you.
3 min read
Test state, tools, and repeated execution before release.
Build stateful agent tests with fictional API fixtures, loop limits and JSON argument checks. Verify side effects instead of trusting a success message.
3 min read
Compare coding, tool-user interaction and general-assistant benchmarks. Understand SWE-bench Verified, τ-bench, τ²-bench and GAIA without mixing their scores.
2 min read
A working playbook for agent evals that gate merges: what to test, how many trials to run, which assertions to trust, and how to keep CI from going flaky.
4 min read
Probe untrusted inputs, disclosure, and agent permissions in authorized test environments.
Test direct and indirect injection at tool boundaries, score unauthorized effects and preserve legitimate tasks. Includes a shared-state fixture matrix.
3 min read
Map the OWASP 2025 LLM Top 10 to model tests and application checks. Test Promptfoo PII and extraction plugins without claiming certification.
2 min read
Choose Promptfoo, PyRIT or Inspect AI for the right red-team workflow. Bound cost, isolate tools and score benign utility alongside attack outcomes.
2 min read
Test direct and indirect prompt injection where your agent reads untrusted text, measure utility and attack success together, and cover cases where the user also acts.
3 min read