🏆 ClawBench — Web Agent Benchmark
Can AI agents complete everyday online tasks? ClawBench scores agents on real, live websites (booking flights, ordering groceries, submitting job applications). Two corpora: V1 — 153 tasks across 144 websites · V2 — 130 newer tasks across 63 platforms. Every run is graded twice: a deterministic HTTP-request interception check (Stage 1, the sort key) — then an LLM judge on the intercepted payload (Stage 2 = Reward).
📖 Paper · 💻 GitHub · 🗂 Dataset · 🎞 Traces V1 · 🎞 Traces V2 · 🌐 Site
Intercepted (Stage 1) = agent's final HTTP request matched the task's URL/method schema — deterministic, no judge. Reward (lenient) (Stage 2, headline metric, default sort key) = judge confirms the intercepted payload fulfilled the instruction under the default rubric (no explicit contradiction → match). Reward (strict) = same judge (default deepseek/deepseek-v4-pro) under the stricter rubric (ambiguous → mismatch), shown for ablation. Rows are ranked by Reward (lenient) DESC, then Intercepted DESC as tiebreak. V2 is Hermes-only; alternative harnesses are evaluated separately. Partial = batch attempted fewer than the full corpus (mid-run abort / queue cap); rates are over attempted, not over corpus.
1 | 2 | 3 |
|---|---|---|
About ClawBench
Why a new benchmark?
Existing browser-agent benchmarks either run on synthetic / sandboxed websites (WebArena, VisualWebArena) or only check whether the agent reached the endpoint (WebVoyager). ClawBench runs on live, real-world websites and verifies the payload the agent submitted — so an agent that types the wrong delivery address into Uber Eats fails, even if its last HTTP request hit the correct endpoint.
Two corpora
- V1 — 153 tasks across 144 real websites (the paper).
- V2 — 130 newer everyday tasks across 63 platforms, expanded coverage of e-commerce / form-filling / authentication-walled flows.
Two-stage scoring
| Stage | What it checks | Output |
|---|---|---|
| 1. Interception | Did the final HTTP request match the task's URL + method + canonical body schema? | intercepted ∈ {true, false} |
| 2. Judge | Given the natural-language instruction and the intercepted payload, did the agent submit the right thing? | match ∈ {true, false, null} |
Reward = Intercepted ∧ Match. Full prompt + judge model details: eval/scoring.md ↗
What ships with every run
A 5-layer trace bundle (downloadable from the Traces datasets above):
recording.mp4— full browser session videoactions.jsonl— every click / type / scrollagent-messages.jsonl— model inputs & outputs (incl. reasoning)requests.jsonl— every HTTP request the page madeinterception.json— graded final requestrun-meta.json— model, harness, scores, timing
Reproducing
pip install clawbench-eval
clawbench run --model <your-model> --harness hermes --corpus v2
python scripts/clawbench_rescore.py --judge-model deepseek-v4-pro --only-batch <your-batch-dir>
🚀 Submit your model
Submissions are accepted as PRs to the leaderboard CSV in the dataset repo:
Open the CSV in the dataset repo ↗
Required steps
- Run the benchmark — install
pip install clawbench-eval, thenclawbench run --model <your-model> --harness hermes --corpus v2(orv1). Use the included harnesses (hermes / openclaw) so traces follow the standard 5-layer format. - Score —
python scripts/clawbench_rescore.py --judge-model deepseek-v4-pro --only-batch <your-batch-dir>producesrescore-summary.jsonwith the cells you'll need. - Upload traces (recommended) — push the 5-layer run bundles to
TIGER-Lab/ClawBenchV2Trace(orNAIL-Group/ClawBenchV1Trace) so others can audit. - Open a PR — add one row per
(model, harness, corpus)toleaderboard/results.csvwith columns:model,harness,dataset,passed,total,pass_rate,reward_rate,wall_hours. Link the trace bundle in the PR description.
We re-run a sample of your submitted traces with our judge before merging — to keep the table honest.
For step-by-step instructions, see eval/scoring.md.