AI Testing & Benchmarking

Distilling the human experience. The future is already here, just unevenly distributed.

ARC Prize is a $1,000,000+ nonprofit, public competition to beat and open source a solution to the ARC-AGI benchmark.

Traces, evals, prompt management and metrics to debug and improve your LLM application. Integrates with Langchain, OpenAI, LlamaIndex, LiteLLM, and more.

Terminal-bench is a collection of tasks and an evaluation harness to help agent makers quantify their agents' terminal mastery.

Find the best AI for you. Compare answers across top AI models, share your feedback and power our public leaderboard.

AI Trading Benchmark. The first benchmark designed to measure AI's investing abilities. Watch AI models trade with real capital.

Comparison and analysis of AI models and API hosting providers. Independent benchmarks across key performance metrics including quality, price, output speed & latency.

METR is a research nonprofit which develops and runs cutting-edge tests of the capabilities of general-purpose AI systems.

Testing platform for AI models. Gain control over AI risks with a holistic testing platform for Quality, Security & Compliance for all AI models, from tabular ML models to LLMs.

A benchmark for LLMs designed with test set contamination and objective evaluation in mind.

An open-source research project developed by members from LMSYS and UC Berkeley SkyLab. Our mission is to build an open crowdsourced platform to collect human feedback and evaluate LLMs under real-world scenarios.

Holistic Evaluation of Language Models (HELM) is a living benchmark that aims to improve the transparency of language models.

ML metadata store. Manage all your model-building metadata in a single place. Track experiments, register models, integrate with any MLOps tool stack.

Open and responsible development of LLMs for code. BigCode is an open scientific collaboration working on the responsible development of large language models for code

WandB is a central dashboard to keep track of your hyperparameters, system metrics, and predictions so you can compare models live, and share your findings.

SuperGLUE is a new benchmark styled after original GLUE benchmark with a set of more difficult language understanding tasks, improved resources, and a new public leaderboard.