API Design¶

This page explains why the public API is shaped the way it is. It is aimed at contributors and at anyone extending the framework beyond what the TOML config exposes. For the exact signatures, follow the links into the API Reference.

The pipeline contract¶

Everything in APE is organized around one fixed pipeline:

DataModel ─► prompt ─► Solver ─► SolutionAdapter ─► Checker ─► CheckResult

The framework owns the orchestration of this pipeline; the benchmark author owns three typed contracts on it:

Nothing else is mandatory. The model backend, tools, concurrency, persistence, and the dashboard are all configuration on top of these three contracts.

Generics carry the types end to end¶

The two type variables DataT and SolutionT thread through the entire stack. A Solver[DataT, SolutionT] produces a SolutionT, the SolutionAdapter[SolutionT] constructs it, and the Checker[DataT, SolutionT] consumes both — so the data schema and the solution type are checked for consistency across the pipeline rather than degrading to Any at the seams.

DataProvider[DataT] ─► Solver[DataT, SolutionT] ─► SolutionAdapter[SolutionT]
                                                          │
                          Checker[DataT, SolutionT] ◄─────┘

The code uses PEP 695 syntax throughout (class Solver[DataT: DataModel, SolutionT]:), which is why Python 3.12 is the floor.

Two return-value contracts¶

The pipeline has exactly two places where a stage reports an outcome, and each has a small, closed result type rather than exceptions or sentinel values:

• The solver returns a Result[SolutionT] — a closed union defined in ape/llm/utils.py as type Result[T] = Ok[T] | Err, where Ok[T] is a frozen dataclass wrapping the parsed solution (val: T) and Err is a frozen dataclass wrapping an ErrKind enum (GAVE_UP, PARSING_FAILED, INTERNAL_ERROR). This lets the orchestrator pattern-match on the outcome and decide whether checking even makes sense.
• The checker returns a CheckResult — Passed or Failed(reason), each optionally carrying a runtime (Optional[timedelta], not a raw number — pick that up in any custom Report) and a free-form metadata: dict[str, Any]. Failed.reason is a string because it becomes a class label in the visualizer's breakdown; distinct reasons become distinct bars.

Because both are closed unions, downstream code (saving, visualizing, the dashboard) handles every case explicitly. A solver Err means no CheckResult is produced at all, which the OutputBatch records as None.

Configuration validates up front¶

SolverConfig and RunParams are frozen dataclasses, and SolverConfig does all its validation in __post_init__ (non-empty prompts, a callable adapter, a compaction threshold in (0, 1], a positive step limit, a non-empty backoff schedule, …). The orchestrator builds one SolverConfig and constructs a fresh Solver per worker from it via Solver.from_config.

The design goal is fail fast, fail before the dashboard: a misconfigured run should raise during construction, not twenty minutes into a long evaluation. The config-driven shell (ape.run) reinforces this by building and validating every component — data provider, checker, solver config, run params, hooks — before the CLI ever takes over the terminal.

Hooks are validated like everything else at startup (they must resolve to a callable), but their runtime errors follow the opposite policy: exceptions raised by on_batch_saved are logged and swallowed. The reasoning is that a hook is a side-channel (mirror to an external store, push a metric); letting a flaky external service abort the run — or worse, lose an already-saved batch — would invert the priority. Hooks should not be load-bearing.

Components are injected, never imported by the core¶

The framework core contains no problem-specific imports. Instead, the TOML shell resolves components by dotted path at runtime through _import_dotted_path, raising ConfigError with an actionable message when a path is wrong:

[data]
schema = "problems.cubes.CubeProblem"

[adapter]
callable = "problems.cubes.parse_solution"

[checker]
callable = "problems.cubes.CubeChecker"

This keeps the dependency arrow pointing one way — your benchmark depends on ape, never the reverse — and means a new benchmark is a new package plus a TOML file, with no edits to the framework.

Factories, not instances¶

Two extension points are passed as factories rather than ready-made objects, because the runner needs to create one instance per worker thread:

• CheckerFactory = Callable[[], Checker]. The CheckerOrchestrator calls it once per checker worker and keeps the instances in a Queue "carousel", checking one out per request and returning it afterward. A factory (not a shared instance) lets each worker own non-thread-safe state — a database connection, a subprocess, a cache.
• Tool factories, normalized by _instantiate_tools. A tool entry's callable may return a single BaseTool, a list of them, or a ToolProvider — a tiny Protocol with get_tool(). The protocol exists so tools that need configuration or state (a connection pool, a server address) can take it through their constructor and still present the uniform "give me the tool(s)" interface.

The checker has two modes — verdict, and optionally an in-loop tool¶

By default the Checker is the deterministic verifier at the end of the pipeline: the solver produces a solution, the orchestrator hands it (with the data item) to the CheckerOrchestrator, and the resulting CheckResult lands in the OutputBatch. The benchmark author implements one check(data, solution) -> CheckResult method and the framework takes care of the rest.

The same Checker can additionally be exposed to the LLM as a tool during the solve, by setting in TOML:

[checker]
callable      = "myproj.MyChecker"
expose_as_tool = true
checker_tool  = "myproj.make_checker_tool"   # required iff expose_as_tool

The checker_tool factory wraps the checker into a BaseTool that the LLM can call mid-solve to test candidate answers before committing to a final one. Two consequences worth knowing:

• The "verdict at the end" path runs regardless of whether the checker was also exposed as a tool — the saved CheckResult is always the final invocation, not a mid-solve one.
• The two surfaces share a Checker instance per worker (the carousel), so a stateful checker (e.g. one holding a database handle) sees both kinds of calls and must remain thread-safe with respect to itself.

This is why the design separates "the Checker contract" (the typed verifier) from "exposing the checker" (a config-level decision): the contract stays the same in either mode, only the call sites differ.

OutputBatch is the unit of work and of persistence¶

Because runs_per_data can be greater than one, the natural unit is not a single attempt but all attempts for one data item: an OutputBatch. It holds the input DataModel plus parallel lists of solver results and (optional) check results, and it knows how to pickle() / from_pickle() itself via jsonpickle. The runner produces one batch per data item, the SaveHandler writes one JSONL line per batch, and the visualizer reads them back. Keeping the batch as the atomic unit is what makes "resume" and "combine sharded runs" well-defined operations.

The identifier system¶

Every DataModel designates exactly one field as its identifier. The default is a field literally named id; any other field can be promoted with id_field(), a drop-in for pydantic.Field() that tags the field via metadata. __pydantic_init_subclass__ enforces "exactly one identifier" at class-definition time.

Two class attributes fall out of this and matter elsewhere:

• id_field_name — the Python attribute name, used to read data.data_id.
• id_key — the serialized key (the field's alias if it has one), used when scanning the raw input file in DataProvider.count() without parsing every record.

The identifier is the join key for the whole framework: it labels dashboard rows, names sessions (s{solver}-d{data}-r{run}), and is the resume key the save handler tracks.

Telemetry is an observer, not a dependency¶

The runner reports to the outside world through a single RunObserver protocol with two methods: on_progress and on_worker_status. This inverts the dependency — the runner doesn't know about the rich dashboard; it knows about an interface the CLI happens to implement. Two consequences:

• A headless run (or a test) passes no observer and gets a NullObserver, which drops every event. The runner's code path is identical either way, so "no UI" is never a special case.
• Worker status is batched through a background pump thread (WorkerStatusTracker) so high-frequency status updates from many workers never block solving, and a misbehaving observer is swapped for a NullObserver rather than being allowed to crash the run.

Concurrency model¶

A run is multi-threaded, with two thread pools fed by queues:

• The SolverOrchestrator owns a pool of solver_workers threads. Each pulls data items off an input queue, runs runs_per_data sessions, and pushes a finished OutputBatch onto an output queue. The main thread drains that queue to save results.
• The CheckerOrchestrator owns a pool of checker_workers and the checker carousel described above. A solver worker calls into it synchronously per solution. Implementation note: the call is executor.submit(...).result(timeout=...) in a loop that re-checks self._cancel.is_set() between polls, so a long check() cannot delay shutdown — the same cooperative-cancellation pattern used everywhere else.

Shutdown and interruption are cooperative. Each orchestrator owns its own threading.Event (self._cancel), and Runner.run calls stop() on both in its finally block, which sets the events and shuts down the underlying ThreadPoolExecutor. The solver orchestrator propagates its event to each Session (session.cancel), so solver workers check session.cancelled() between steps and raise InterruptedError to unwind cleanly. Sentinels (None) on the data queue wake idle workers so the pools can drain on stop. The orchestrators also re-raise any worker exception on the main thread (_check_futures) instead of letting it vanish inside a future.

Persistence and resume¶

The SaveHandler is built for crash safety. Each batch is appended to the JSONL results file and the run metadata is rewritten, both flushed with os.fsync. Metadata carries a save_format_version; loading a save written by a newer format version is refused rather than silently mis-parsed.

Resume is the natural consequence of streaming results plus tracking completed ids: on startup with load_save enabled (the default), the handler returns the set of completed ids, and the runner skips any incoming data item already in it. Note the deliberate split of responsibilities — range selection (start_id / end_id on the DataProvider) decides which items exist for this run, while resume decides which of those to skip because they're already done. They compose without interfering.

Where to extend¶

The solve loop itself — the part that turns a prompt into a parsable answer — is the one piece with enough internal structure to warrant its own page; see the LLM Backend Design.