# The benchmark — protocol
What the site's numbers come from (`src/content/benchmark.json`, shown on the home page and the Why AI page). A
number the site shows is a median of runs made by this protocol, never an estimate; until there are runs, the
site shows `[X]`.
## The question
How much does it cost an AI agent to build the same feature on MMOFramework, on Laravel + Vue, and on
Symfony + Twig — and does what it builds work?
## The feature
[SPEC.md](SPEC.md): an invoices page with a PDF export. One text, given word for word to every run, written
without naming any framework's classes.
## The runs
| | |
|---|---|
| Agent | Claude Code, the same model and effort for every run, recorded in the results |
| Stacks | MMOFramework (a project made by `php tools/mmo.php project`, its MCP server connected); Laravel + Vue (`laravel new`, Vue with Vite); Symfony + Twig (`symfony new --webapp`) |
| Starting point | a freshly created project of the stack, committed, its own test suite green, nothing else |
| Help | what the stack ships with for agents and nothing more: MMOFramework's AGENTS.md, skills and MCP server; the others' own documentation files if their installer writes some. No hand-written notes for any of them |
| Prompt | SPEC.md, as is, then nothing: the run ends when the agent says it is done or after 90 minutes |
| Runs | 5 per stack, each in a fresh copy of the starting point, in alternating order |
## What is measured
From each run's transcript (`scripts/import-session.php` reads the same file the site replays):
| Measure | From |
|---|---|
| tokens | the sum of every request's input (cache reads included) and output tokens |
| minutes | the first prompt to the last answer |
| turns | the assistant's requests |
| errors fixed alone | each check that failed (tests, analysis, a page answering 500) and passed later in the run without help |
Then, by hand, on the finished project, with a checklist taken from SPEC.md:
| Measure | |
|---|---|
| done | every point of the spec's "Done means" holds (yes/no, and which ones fail) |
| tests | its test suite passes |
## What is published
- `src/content/benchmark.json`: every run (stack, run, model, tokens, minutes, turns, errors fixed, done, tests),
and the medians per stack. The site computes what it shows from it.
- every transcript, as JSON, beside the results (`docs/benchmark/runs/`), so anyone can read what the agent did.
- this protocol, at the version the runs followed.
A run that did not finish (90 minutes, or the agent gave up) is published and counted: it is not done, and its
tokens are what it spent.