Core Concepts¶

This page introduces the core pieces you will see in every APE benchmark and explains how they fit together. Each concept maps directly to a config section or a small bit of code you implement.

The evaluation pipeline¶

At a high level, APE turns problem data into model answers and then checks them with a deterministic verifier:

data -> prompt -> solver -> adapter -> checker -> output

The runner orchestrates the whole pipeline, including parallel workers, saving progress, and logging.

Data model and input files¶

DataModel A schema (Pydantic model) that defines one problem instance. One field is the identifier used across the run for logging, saving, and resume. The default identifier is a field named id, or you can mark another field with id_field().

DataProvider Loads the input file (.csv or .jsonl) and yields DataModel instances. The optional start_id and end_id parameters slice the input range by file order — processing begins at the item matching start_id and stops after the item matching end_id (both inclusive), regardless of how the id values compare. This is distinct from resume: skipping already-completed items on restart is handled by the save handler (see Runner, workers, and saving).

Prompt template¶

The instruction prompt is a mustache template that APE renders per data item. Fields of the DataModel are available as {{ field_name }}. If a field declares a Pydantic alias, the alias is the variable name in the template, not the Python field name. The rendered prompt is sent to the solver for that item.

Solver sessions¶

Solver The solver owns the LLM call sequence, including optional tools and retries. It can run multiple steps before producing a final answer.

Session Each attempt to solve one data item is a session. The session captures prompts, LLM responses, tool calls, and failures into a transcript (see Outputs and logs). If runs_per_data > 1, the runner creates multiple sessions per item.

Solution adapter¶

SolutionAdapter A small function (str) -> SolutionT that parses the raw model output into a structured solution type. If parsing fails, the solver can reformat the answer with a dedicated prompt and retry a limited number of times (controlled by solution_adaption_attempts).

Checker¶

Checker A deterministic verifier that receives the data item and the parsed solution. It returns either Passed or Failed plus optional runtime and metadata. Checkers are pure and fast by design: they validate, they do not solve.

Tools¶

Tools are LangChain BaseTool instances that the solver can call between LLM steps. APE loads them from the [tools.X] config sections and binds them to the model automatically. A tool factory callable may return a single BaseTool, a list, or a ToolProvider instance (a lightweight protocol with a get_tool() method).

You can also expose a checker as a tool by setting expose_as_tool = true and checker_tool = "..." in the [checker] config section. This lets the solver call the verifier mid-session before committing to a final answer.

Runner, workers, and saving¶

Runner Coordinates the pipeline. It creates solver and checker workers, schedules data items, and persists progress via the save handler.

Run parameters solver_workers, checker_workers, and runs_per_data control concurrency and sampling. save_file and metadata_file define output locations. If load_save is enabled (the default), the save handler reads completed ids from metadata_file on startup and skips them — this is how a run resumes after interruption.

Outputs and logs¶

Every run produces a small set of artifacts:

• output/results.jsonl: per-item batches of solver and checker results
• output/metadata.json: run metadata, including the solver config
• output/run.log: CLI log output
• output/sessions.log: per-session transcripts in human-readable text
• output/sessions.jsonl: the same transcripts in JSON Lines format, useful for automated analysis

The visualizer can turn the saved results into a summary chart.

Configuration as the glue¶

APE is configured by a single TOML file that connects all of the above pieces:

• data file and schema
• prompt template
• model provider and name
• solution adapter
• solver options and tools
• checker implementation
• run and CLI settings
• hooks (e.g. on_batch_saved callback)

See the TOML configuration reference for the full schema. When you are ready for more depth, continue to the user guide pages on running a benchmark, LLM integration, and CLI.