One engine.
Now repair it.

1Install

Get the repository. The environment is in env/. Nothing else to install: no pip install, no compiler, no browser. Check python3 --version: if it is not 3.12.x, write python3.12 wherever these steps say python3. The steps use /ABS/REPO for your clone, /ABS/RUNS for a folder for run records outside the clone, and /ABS/WORK for an empty folder outside the clone where the agent’s client runs.

git clone https://github.com/ToolsEnabled/head-gasket-bench.git /ABS/REPO
cd /ABS/REPO/env
python3 -m headgasket pins --check     # verifies the 19 pinned source files and the 5 configurations
python3 -m headgasket tools            # prints the 18 tools

2Connect your MCP client

One server process is one run: pick a variant and a new folder for the record (or an empty folder of yours that other users cannot write to). Use absolute paths; MCP clients start servers from a working directory you do not control. A start that makes no tool call, such as a client’s health check (claude mcp list) or a session you quit before the agent called a tool, records nothing, so the folder stays free for the run; if a client kills the server before it can clean up, the next start clears what is left. Once the agent has made a call, the folder holds that run, and the next run needs a new folder.

python3 /ABS/REPO/env/bin/headgasket-mcp serve --variant V1 --out /ABS/RUNS/v1-first-try

These are examples of the wiring, not a recommendation of any client or model.

Claude Code

Run both commands in /ABS/WORK, an empty folder outside the clone: claude mcp add registers the server only for the folder it is run in, and --tools "" turns Claude Code’s own tools off, so the agent has only the 18 job tools.

mkdir -p /ABS/WORK && cd /ABS/WORK
claude mcp add headgasket -- python3 /ABS/REPO/tools/claude-code-proxy.py python3 /ABS/REPO/env/bin/headgasket-mcp serve --variant V1 --out /ABS/RUNS/v1-first-try
MAX_MCP_OUTPUT_TOKENS=150000 claude --tools "" --allowedTools mcp__headgasket

or in /ABS/WORK/.mcp.json:

{
  "mcpServers": {
    "headgasket": {
      "command": "python3",
      "args": ["/ABS/REPO/tools/claude-code-proxy.py", "python3",
               "/ABS/REPO/env/bin/headgasket-mcp", "serve", "--variant", "V1",
               "--out", "/ABS/RUNS/v1-first-try"]
    }
  }
}

look returns about 196,000 characters. Claude Code saves an MCP tool result that is larger than the size the tool declares to a file and shows the model only a short preview, and it cuts results above MAX_MCP_OUTPUT_TOKENS (25,000 tokens by default). So start the server through tools/claude-code-proxy.py, which adds that size declaration to each tool and passes every message through unchanged, and start Claude Code with MAX_MCP_OUTPUT_TOKENS=150000. Other clients may cut long results too; check yours before a run.

Keep the agent to the job tools. The repository holds complete correct solutions, so an agent that can read files could copy one. Start the client in an empty folder outside the repository with its own file and shell tools turned off, as the Claude Code commands above do, and say how in your SUBMISSION.json notes. The README shows the full headless command the claude-haiku-5-5 baseline ran.

Codex CLI (~/.codex/config.toml)

[mcp_servers.headgasket]
command = "python3"
args = ["/ABS/REPO/env/bin/headgasket-mcp", "serve", "--variant", "V1",
        "--out", "/ABS/RUNS/v1-first-try"]
startup_timeout_sec = 60
tool_timeout_sec = 600

This file is global, so every Codex session starts the server: remove the entry, or change --out, after the run.

Any other stdio MCP client

Start the same command and exchange one JSON object per line on stdin and stdout (MCP protocol version 2025-11-25). The minimal session:

{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"protocolVersion":"2025-11-25"}}
{"jsonrpc":"2.0","method":"notifications/initialized"}
{"jsonrpc":"2.0","id":1,"method":"tools/list"}
{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"look","arguments":{}}}

3Run the job

The environment carries no prompt; your client supplies one. Record the prompt you used with your run. The server writes the record after every call and exits when the client closes its input. When turns_remaining reaches 0, stop: the launcher answers any further call with the budget refusal and does not record it, so the record stops at 1,024 attempts. It also answers, without recording it, a remove naming a route_id that an installed part does not have (the pinned environment would fail every later call), and lists the part’s routes.

Replace the failed head gasket on the prepared service-stand engine and
restore the assembly and timing. Use the headgasket tools to inspect, look
up service information and do the work. End with finish within the work
budget. Every job-tool call attempt, including a refused action, uses one
environment turn. The remaining budget is shown in tool responses; if it is
exhausted, end your response. Each successful action adds estimated
technician time. Use only the supplied job tools.

4Validate

Copy run.json, actions.jsonl and report.json into a folder runs/submitted/<name>/, add your SUBMISSION.json, and let the repository’s validator check the files, your declaration, secrets and personal paths, the hash chain and the report, that the run is not an exact copy of a published or submitted run (a near copy is marked for the maintainer’s review), the pinned configuration, an exact replay and the independent scorer. Then rebuild the board. A run that stopped without finish is scored as it stands.

Release 0.1.2 corrects the independent scorer’s CP02 to count refused attempts as the paper defines it and the environment decides it, so a run that the 0.1.1 validator refused only because the two disagreed on CP02 now passes (release notes).

cd /ABS/REPO
mkdir -p runs/submitted/<name>
cp /ABS/RUNS/v1-first-try/{run.json,actions.jsonl,report.json} runs/submitted/<name>/
# add runs/submitted/<name>/SUBMISSION.json (SUBMISSIONS.md shows it)
python3 tools/validate_submission.py --write-result runs/submitted/<name>
python3 tools/validate_submission.py runs/submitted/<name>
python3 tools/build_board.py

5Submit

Open a pull request that adds runs/submitted/<name>/ with your run’s record files and a SUBMISSION.json that states the model label, the variant, the client, the date and the submitter (shown on the board), and the rebuilt leaderboard/board.json. A CI validator, run from the main branch’s code, replays the record and rescores it, and a maintainer reviews every submission by hand before merging it. Accepted runs appear on the board with the model label you state. ToolsEnabled cannot verify which model served a run. The steps are in SUBMISSIONS.md.