What's in the benchmark
Built to be running on a single Docker image: agent + judge + LibreOffice + CJK fonts all live in one container.
Real Office Artifacts
xlsx, docx, pptx, pdf, csv, images, source files.
15 Corporate Professions
Marketing, Sales, BA, Operations, Dev, SRE, HR, Finance, PM, Legal, Algorithm, QA, UI/UX, Administration, and General.
LLM-Based Rubric Grading
Output artifacts are checked against expert-annotated rubrics by an LLM judge.
Leaderboard
All models are evaluated with thinking mode enabled. For models that expose a configurable reasoning effort / thinking level, the highest setting is used.
| # | Model | Avg@3 | Pass@3 | Pass^3 |
|---|---|---|---|---|
| loading… | ||||
Quickstart
A full run is three commands once your gateway is configured.
# 1. Configure an OpenAI-compatible gateway (used for both agent and judge). cp openclaw_config/openclaw.json.example openclaw_config/openclaw.json cp openclaw_config/.env.example openclaw_config/.env # 2. Build the single agent+judge Docker image. bash docker/fetch-lo.sh docker build -f docker/Dockerfile.agent -t forte-agent:latest docker/ # 3. Run. bash scripts/run.sh # Or run the benchmark CLI directly with an explicit model. python scripts/benchmark.py \ --model your-gateway/claude-sonnet-4.6 \ --dataset ./data \ --runs 3 \ --concurrency 1 \ --output-dir ./results # Optional overrides. MODEL=your-gateway/claude-sonnet-4.6 RUNS=3 CONCURRENCY=2 bash scripts/run.sh
Resources
GitHub Repository →
Task Definition, Source Code, Docker Recipe, Judge Implementation.