HumanEval
A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
MiniCPM-SALA from OpenBMB currently leads the HumanEval leaderboard with a score of 0.951 across 66 evaluated AI models.
What HumanEval measures
HumanEval is a text benchmark that evaluates large language models on reasoning and code tasks. LLM Stats tracks 66 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.8, with the leader reaching 1.0.
Compare leaders on the best AI for reasoning and best AI for code leaderboards.
Publication
- Paper
- Evaluating Large Language Models Trained on Code
- Authors
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and 54 others
- Published
- arXiv
- 2107.03374
Abstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
MiniCPM-SALA leads with 95.1%, followed by 
Kimi K2 0905 at 94.5% and 
Claude 3.5 Sonnet at 93.7%.
Progress Over Time
Interactive timeline showing model performance evolution on HumanEval
HumanEval Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  OpenBMB | 9B | — | — | ||
| 2 | Moonshot AI | 1.0T | — | — | ||
| 3 | Anthropic | — | — | — | ||
| 4 |  OpenAI | — | — | — | ||
| 5 | Moonshot AI | 1.0T | — | — | ||
| 6 |  Alibaba Cloud / Qwen Team | 32B | — | — | ||
| 7 | OpenAI | — | — | — | ||
| 8 |  Sarvam AI | 30B | — | — | ||
| 9 | Anthropic | — | — | — | ||
| 9 |  Mistral AI | 123B | — | — | ||
| 11 | Alibaba Cloud / Qwen Team | 34B | — | — | ||
| 12 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 13 | 8B | — | — | |||
| 13 | 8B | — | — | |||
| 15 |  Google | — | — | — | ||
| 16 |  Amazon | — | — | — | ||
| 16 |  DeepSeek | 236B | — | — | ||
| 16 | 405B | — | — | |||
| 19 | Mistral AI | 24B | — | — | ||
| 19 |  Meituan | 560B | — | — | ||
| 21 | 70B | — | — | |||
| 21 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 21 |  xAI | — | — | — | ||
| 21 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 25 | Anthropic | — | — | — | ||
| 25 | OpenAI | — | — | — | ||
| 27 | OpenAI | — | — | — | ||
| 28 | Google | 27B | — | — | ||
| 29 | OpenAI | — | — | — | ||
| 30 | OpenAI | — | 128K | $10.00 / $30.00 | ||
| 31 | Alibaba Cloud / Qwen Team | 73B | — | — | ||
| 32 | Alibaba Cloud / Qwen Team | 72B | — | — | ||
| 33 | xAI | — | — | — | ||
| 34 | Google | 12B | — | — | ||
| 34 | Amazon | — | — | — | ||
| 36 | Anthropic | — | — | — | ||
| 37 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 37 | Mistral AI | 24B | — | — | ||
| 39 | Google | — | — | — | ||
| 40 | Alibaba Cloud / Qwen Team | 15B | — | — | ||
| 41 |  Microsoft | 15B | — | — | ||
| 42 | 7B | — | — | |||
| 43 | Mistral AI | 22B | — | — | ||
| 43 | Amazon | — | — | — | ||
| 45 | 70B | — | — | |||
| 46 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 47 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 48 | Anthropic | — | — | — | ||
| 49 | 2B | — | — | |||
| 49 | Google | 8B | — | — |
FAQ
Common questions about HumanEval.
More evaluations to explore
Related benchmarks in the same category
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions