Outputs & Results

Every APE run writes output files to the output/ directory relative to the working directory from which the command is run (or to the directory specified with -o). This page describes each file and how to interpret the summary statistics and chart.

Output files

File Created Contents
results.jsonl Once per run One jsonpickle-encoded OutputBatch per line
metadata.json Once per run, updated after each item Run parameters and progress
run.log Once per run Full log output
sessions.log Once per run Human-readable session transcripts
sessions.jsonl Once per run Same transcripts in JSON Lines format
results_visualized.png End of run Stacked-bar chart

All paths are configurable via [run] and [cli] in the TOML file.

results.jsonl

One line per completed data item. Each line is a jsonpickle-encoded OutputBatch containing:

  • the input DataModel instance
  • a list of solver results — Ok(solution) or Err(kind) — one per runs_per_data session
  • a list of check results — Passed, Failed(reason), or None — one per session; None means the solver errored and there was nothing to check

Read the file with OutputBatch.from_pickle(line) per line, or pass it directly to python -m ape visualize for a summary.

metadata.json

A single jsonpickle-encoded object written at run start and updated after each saved item. Fields:

Field Description
save_format_version Integer version used for forward-compatibility checks
run_params The RunParams dataclass used for this run
solver_config The full SolverConfig (model, adapter, tools, …)
checker_params Keyword arguments passed to the checker factory
start_date Datetime when the run started
last_time Datetime of the last save
total Number of completed items so far
completed_ids Sorted list of all completed data IDs

The completed_ids field is what the save handler reads on restart to determine which items to skip.

sessions.log and sessions.jsonl

Each session appends one record when it closes. The .log file is human-readable:

2026-05-28 12:00:00,000
══════════════════════════════════════════════════════════
SESSION s0-d5-r0
══════════════════════════════════════════════════════════
Data: {'k': 5}

── Turn 0: Prompt ──
text:
[system prompt]

── Turn 0: Prompt ──
text:
Let k be 5. Generate integers x, y such that x^2 + y^2 = k ...

── Turn 1: LLM Response ──
text:
1 2

── Turn 1: Session End ──
Reason: ok
══════════════════════════════════════════════════════════

Each record is prefixed with the log timestamp, and every entry header has the form ── Turn N: <kind> ── (including Session End).

The session ID format is s{solver_id}-d{data_id}-r{run_index}. Each entry has a kind (Prompt, LLM Response, Tool Call, Tool Output, Feedback, Compact Context, Failure, Session End) and a turn counter that increments with each LLM call.

The .jsonl sibling has the same data as one JSON object per line — useful for programmatic analysis of transcripts.

Visualizer

At the end of every run APE automatically runs the visualizer, which prints a summary to the terminal and saves a chart as results_visualized.png next to results.jsonl.

To regenerate the visualization from an existing run:

python -m ape visualize output/
# or point directly at the save file:
python -m ape visualize output/results.jsonl

Summary statistics

The terminal output includes:

  • Data items — total number of completed items
  • Total attempts — data items × runs_per_data
  • Correct / Incorrect — counts and percentages over all attempts
  • ≥ 1 correct attempt — fraction of items where at least one session passed (equivalent to pass@k)
  • All attempts correct — fraction of items where every session passed
  • Class breakdown — attempt counts per outcome class
  • Checker runtime — median and mean seconds per outcome class

Chart

The PNG is a stacked bar chart with one bar per data item. Each segment is one outcome class:

Class Color Meaning
Correct Green Passed check result
<reason> Distinct color per reason Failed(reason) — one class per distinct reason string, colored from a fixed categorical palette
Solver error: gave_up Grey Model output contained I_GIVE_UP
Solver error: parsing_failed Grey Adapter could not parse after all attempts
Solver error: internal_error Grey Unexpected exception

When a run has more than 50 items the chart wraps into multiple rows of 50 bars each.

Interpreting results with multiple runs per item

When runs_per_data > 1, each bar represents several sessions. Use the stacked bar to see whether failures are consistent (same class every run) or noisy (different classes across runs). The summary's "≥ 1 correct" metric is the standard pass@k estimate when k = runs_per_data.