Running a Benchmark¶

This page explains what happens between python -m ape config.toml and the final summary chart: how the runner schedules work, what a solver session is, and which parameters control concurrency and resume behaviour.

Runner overview¶

Runner.run() is the single entry point that drives the whole pipeline. It builds two orchestrators and then feeds data items through them:

DataProvider
    │
    ▼
SolverOrchestrator  ──►  CheckerOrchestrator
    │                           │
    └───────── OutputBatch ─────┘
                    │
                    ▼
              SaveHandler

The runner feeds every item from the DataProvider into the SolverOrchestrator. When all sessions for an item are done and checked, the SolverOrchestrator emits an OutputBatch for that item. The runner collects batches one by one, passes each to the SaveHandler, and advances the progress counter.

Solver session lifecycle¶

For each data item the solver orchestrator runs runs_per_data sessions. Each session is an independent attempt to solve that item and goes through the following phases:

1. Prompt rendering — the instruction template is filled with the data item's fields and sent to the LLM as the first human message.
2. LLM steps — the solver calls the model, handles tool calls, and loops until it decides an answer is ready or the max_steps limit is reached. When the limit is hit the framework sends a force-finish prompt asking the model for a final answer before moving to solution parsing. The forced answer is parsed like any other, so it can end as Ok(solution), PARSING_FAILED, or GAVE_UP (if it still contains I_GIVE_UP).
3. Solution parsing — the raw model output is passed to the SolutionAdapter. The adapter gets solution_adaption_attempts total parse attempts: the first on the original output, then up to solution_adaption_attempts − 1 times after a reformat prompt. If all attempts fail the session ends with PARSING_FAILED.
4. Checking — a successful parse hands the solution to the CheckerOrchestrator, which runs the deterministic verifier and returns Passed or Failed.
5. Session close — the session appends a SESSION_END entry and flushes the transcript to sessions.log and sessions.jsonl.

A session ends with one of four solver outcomes:

All runs_per_data sessions for an item are collected into a single OutputBatch before the item is considered done. If the run is interrupted mid-item, that item's partial progress is not saved.

Concurrency and workers¶

solver_workers and checker_workers control how many threads run in parallel.

Solver workers each own their own Solver instance (including any stateful tools). They pull items from a shared queue and independently run all runs_per_data sessions for each item before moving to the next one.

Checker workers are kept in a carousel: a pool of Checker instances shared across all solver workers. Each checker is returned to the pool after it finishes so another solver can use it. Checkers are expected to be fast and stateless, so one checker worker is usually enough.

A few practical rules of thumb:

• Set solver_workers to match your rate-limit headroom or the number of CPU cores if running a local model.
• Increase checker_workers only if your checker is slow (e.g. calls an external solver).
• runs_per_data > 1 is useful for measuring pass@k or averaging out non-deterministic model behaviour.

Resume and save handler¶

APE saves results incrementally so a run can be resumed after interruption.

On start, if load_save = true (the default), the SaveHandler reads metadata.json and collects the set of already-completed item IDs. The runner skips those items when feeding from the DataProvider.

After each item, once all its sessions and checks are done, the runner appends the OutputBatch to results.jsonl and updates completed_ids in metadata.json. Both writes are flushed and synced to disk before moving on.

On restart, the runner picks up exactly where it left off: already-saved items are skipped, and any item that was in progress when the run died is retried from scratch.