Developer Guide¶

This guide is for people who want to understand or extend the framework itself, rather than just configure and run an existing benchmark. If your goal is to build a benchmark, the Core Concepts and Building a Custom Benchmark pages are the right starting point; come back here when you need to change APE's internals.

What you'll find here¶

• Development Setup — install the dev toolchain, run the test suite, the formatting / type-checking gates, and build the docs locally.
• Internals
  • API Design — the public API surface, the generic type contracts (DataT / SolutionT), and the extension points that hold the pipeline together.
  • LLM Backend Design — how a single solve is driven as a LangGraph state machine, including tools, context compaction, retries, and the force-finish / reformat fallbacks.
• API Reference — the per-module reference generated from docstrings.

Architecture at a glance¶

A run is a pipeline that turns problem data into verified outcomes:

DataModel -> prompt -> Solver -> SolutionAdapter -> Checker -> CheckResult

Each stage is a separate, independently replaceable component, and the Runner is the only piece that knows about all of them. The framework splits cleanly into three layers:

1. User-implemented components. A DataModel schema, a SolutionAdapter, and a Checker. These are the only parts a benchmark author must write; everything else is configuration.
2. The framework core. The Solver and its LangGraph workflow, the solver/checker orchestrators, the SaveHandler, and the Runner that coordinates them across worker threads.
3. The config-driven shell. ape.run reads a TOML file, imports the components it names, validates everything into a frozen SolverConfig and RunParams, and hands off to Runner.run. The CLI is the live RunObserver attached during the run; the Visualizer and Report consume the saved results after the fact.

Source layout¶

The package mirrors that structure:

Key design choices¶

A few decisions recur throughout the codebase and are worth knowing up front:

• Configuration is validated up front. SolverConfig and RunParams are frozen dataclasses; SolverConfig does all its checks in __post_init__, and the TOML shell (ape.run) validates the surrounding config (unknown keys, file paths, dotted imports) before constructing them. A misconfigured run fails before the dashboard comes up rather than mid-run.
• Components are injected by dotted path, not hard-coded. The TOML shell imports your schema, adapter, checker, and tools by their fully qualified names. The core never imports problem-specific code.
• The solve loop is a graph, not a function. Each session runs a small LangGraph state machine, which keeps "think", "use tools", "compact context", and "finish" as separate nodes with explicit transitions.
• Results are streamed to disk as they complete. The SaveHandler fsyncs every batch and records completed ids, so interrupting a run loses nothing and restarting it resumes automatically.

See API Design for the reasoning behind these in detail.