GPQA
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
Claude Mythos Preview from Anthropic currently leads the GPQA leaderboard with a score of 0.946 across 223 evaluated AI models.
What GPQA measures
GPQA is a text benchmark that evaluates large language models on reasoning, general, physics, biology, and chemistry tasks. LLM Stats tracks 223 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.7, with the leader reaching 0.9.
Compare leaders on the best AI for reasoning, best AI for general, best AI for physics, best AI for biology and best AI for chemistry leaderboards.
Publication
- Paper
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Authors
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, and 4 others
- Published
- arXiv
- 2311.12022
Abstract
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.
Claude Mythos Preview leads with 94.6%, followed by 
Gemini 3.1 Pro at 94.3% and Claude Opus 4.7 at 94.2%.
Progress Over Time
Interactive timeline showing model performance evolution on GPQA
GPQA Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | Anthropic | — | — | — | ||
| 2 | Google | — | 1.0M | $2.50 / $15.00 | ||
| 3 | Anthropic | — | 1.0M | $5.00 / $25.00 | ||
| 4 | Anthropic | — | 1.0M | $5.00 / $25.00 | ||
| 4 |  OpenAI | — | 1.1M | $5.00 / $30.00 | ||
| 6 | OpenAI | — | — | — | ||
| 7 | OpenAI | — | 1.0M | $2.50 / $15.00 | ||
| 8 |  Alibaba Cloud / Qwen Team | — | 1.0M | $1.25 / $3.75 | ||
| 8 | OpenAI | — | 400K | $1.75 / $14.00 | ||
| 10 | Google | — | — | — | ||
| 11 | Anthropic | — | 1.0M | $5.00 / $25.00 | ||
| 12 |  Moonshot AI | 1.0T | 262K | $0.95 / $4.00 | ||
| 13 | Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 13 | Google | — | 1.0M | $0.50 / $3.00 | ||
| 15 |  DeepSeek | 1.6T | 1.0M | $1.74 / $3.48 | ||
| 16 | Anthropic | — | 200K | $3.00 / $15.00 | ||
| 17 |  Meta | — | — | — | ||
| 18 |  ByteDance | — | — | — | ||
| 19 |  xAI | — | — | — | ||
| 19 | Alibaba Cloud / Qwen Team | 397B | 262K | $0.60 / $3.60 | ||
| 21 | OpenAI | — | 400K | $1.25 / $10.00 | ||
| 21 | OpenAI | — | — | — | ||
| 21 | OpenAI | — | — | — | ||
| 21 | OpenAI | — | — | — | ||
| 21 | OpenAI | — | 400K | $1.25 / $10.00 | ||
| 21 | DeepSeek | 284B | 1.0M | $0.14 / $0.28 | ||
| 27 | OpenAI | — | 400K | $0.75 / $4.50 | ||
| 28 | Alibaba Cloud / Qwen Team | 28B | 262K | $0.60 / $3.60 | ||
| 29 | Moonshot AI | 1.0T | — | — | ||
| 30 | xAI | — | — | — | ||
| 31 | OpenAI | — | — | — | ||
| 32 | Anthropic | — | — | — | ||
| 33 | Google | — | 1.0M | $0.25 / $1.50 | ||
| 34 | Alibaba Cloud / Qwen Team | 122B | 262K | $0.40 / $3.20 | ||
| 35 | — | — | — | |||
| 36 |  Zhipu AI | 754B | 200K | $1.40 / $4.40 | ||
| 37 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 38 | Zhipu AI | 358B | — | — | ||
| 38 | OpenAI | — | — | — | ||
| 38 | xAI | — | 2.0M | $0.20 / $0.50 | ||
| 41 | OpenAI | — | 400K | $5.00 / $30.00 | ||
| 42 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.30 / $2.40 | ||
| 43 | ByteDance | — | — | — | ||
| 44 |  Baidu | — | — | — | ||
| 45 | Anthropic | — | — | — | ||
| 46 | xAI | — | 128K | $3.00 / $15.00 | ||
| 46 |  Microsoft | — | — | — | ||
| 48 | Moonshot AI | 1.0T | — | — | ||
| 49 | Google | 31B | 262K | $0.14 / $0.40 | ||
| 50 | Microsoft | 1.0T | — | — |
FAQ
Common questions about GPQA.
More evaluations to explore
Related benchmarks in the same category
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.