Advanced LLM Run

The Quickstart shows a minimal single-step run. This tutorial covers everything beyond that: multi-step solver sessions, attaching tools, parallel workers, partial dataset slices, and resuming interrupted runs.

Prerequisites

Multi-step solver

By default the solver asks the LLM once and takes its response as the final answer. For hard problems you want the model to reason across multiple steps — calling tools, revising its answer, and only committing when it is ready.

Set max_steps in your config:

[solver]
reformat_prompt = "That answer did not parse. Reply with ONLY 'TRIPLE(x, y, z)'."
max_steps = 20

Each "step" is one LLM inference call. The solver keeps running until either:

  • The model produces a final text response (no tool call) — that response is fed to the adapter.
  • max_steps is reached — the solver force-finishes and tries to parse whatever the model last said.

For pure search problems, a higher max_steps gives the model more room to explore. A value of 1 reproduces the single-shot quickstart behavior.

Reformat retries

When the adapter fails to parse the model's answer (raises any exception), APE shows the LLM a reformat prompt and retries. Control how many times this can happen:

[solver]
reformat_prompt = "That answer did not parse. Reply with ONLY 'TRIPLE(x, y, z)'."
solution_adaption_attempts = 3

Reformat retries are extra LLM calls that happen inside the parsing phase — they do not count toward max_steps. The session is marked PARSING_FAILED if all retries are exhausted.

Attaching tools

Tools are LangChain BaseTool instances that the solver can call between LLM steps. APE loads them from [tools.X] sections and binds them to the model automatically.

In-process tools

These run inside the APE process — no containers needed.

SessionNotebook is a scratchpad the LLM can read and write. Useful for accumulating partial results across steps:

[solver]
tools = ["notebook"]

[tools.notebook]
callable = "ape.tools.SessionNotebook"

Julia lets the solver execute Julia code in a sandboxed server (see Containerized tools below for the server setup):

[solver]
tools = ["julia", "notebook"]

[tools.julia]
callable = "ape.tools.Julia"
response_timeout = 135
execution_timeout = 120

Every key in [tools.X] other than callable is passed as a keyword argument to the tool's constructor, so you can tune timeouts and pool sizes per problem.

Checker as a tool

If you have implemented expose_as_tool = true in your [checker] section (as shown in Building a Custom Benchmark), the solver can call the verifier mid-session. No additional entry in [solver] tools is needed — APE wires it up automatically when expose_as_tool = true.

[checker]
callable = "checker.SOCChecker"
expose_as_tool = true
checker_tool = "tool.soc_checker_tool"

Containerized tools

Some tools run heavy or sandboxed workloads in their own container. APE acts as a client; you bring the container up separately with Docker Compose.

The Julia execution server is an example. You need two matching pieces:

config.toml — client side:

[tools.julia]
callable = "ape.tools.Julia"
response_timeout = 135
execution_timeout = 120

compose.yaml — server side:

services:
  tool-julia:
    image: aboguszewski/julia-exec-server:0.2.3
    profiles: ["julia"]
    ports:
      - "8081:8080"   
    environment:
      SERVER_IP: "0.0.0.0"
      SERVER_PORT: "8080"
      WORKER_COUNT: "1"

Start the server before running APE:

docker compose --profile julia up -d
python -m ape config.toml
docker compose --profile julia down

The profile mechanism lets you start only the services a particular run needs. Use --profile "*" to start every service at once.

Long tool outputs and error summarization

Tool outputs can be very long. When an error output exceeds a character threshold, APE summarizes it with a helper LLM call before including it in the conversation, keeping the context window under control:

[solver]
error_summarization_threshold = 500

To disable summarization entirely, set the threshold higher than any error you expect to encounter (e.g. a very large number).

Context window management

For long multi-step sessions the conversation history can grow close to the model's context limit. APE monitors occupancy and compacts the history when a threshold is crossed:

[solver]
context_window_size = 128_000
context_compaction_threshold = 0.8

context_window_size tells APE the model's limit in tokens. If omitted, APE tries to detect it from the model's metadata. context_compaction_threshold is the fraction of the window that triggers compaction (default 0.8 = 80%). Compaction summarizes the oldest messages, preserving the system prompt and the most recent exchanges.

Running on a dataset slice

For a large dataset you often want to run a subset — during development, for parallel sharding across machines, or to rerun specific items. Use start_id and end_id in the [data] section:

[data]
path = "data/problems.csv"
schema = "data.SOCSchema"
start_id = "100"
end_id   = "200"

APE scans the dataset file sequentially: it starts including rows once it encounters the row whose identifier equals start_id, and stops after the row whose identifier equals end_id. Both bounds are inclusive. The order of rows in the file determines what gets processed — there is no sorting.

Scaling up with workers

Each problem item and each checker call can run in its own worker. Increase concurrency in [run]:

[run]
solver_workers  = 4
checker_workers = 2

solver_workers controls how many items are solved in parallel. Each worker drives its own LLM call sequence independently, so you can saturate a model's API concurrency limit.

checker_workers controls how many checker calls run in parallel after solving. For a fast in-process checker like SOCChecker a value of 1 is fine. For a checker that contacts a remote service (like the Oscar-backed IGP checker), increasing this can meaningfully reduce wall-clock time.

Repeated runs per item

To collect multiple independent solutions for the same problem:

[run]
runs_per_data = 3

Each item gets runs_per_data independent solver sessions. Useful for measuring variance or pass@k metrics.

Resuming an interrupted run

APE saves progress to output/results.jsonl as it goes. If a run is interrupted, restart it with the same config — APE reads the save file on startup and skips already-completed items:

[run]
save_file  = "output/results.jsonl"
load_save  = true

load_save = true is the default. Set it to false to start fresh even if a save file exists.

Putting it all together

A realistic config for a hard search problem with multi-step solving, tools, parallel workers, and resume support:

[data]
path     = "data/problems.csv"
schema   = "data.SOCSchema"
start_id = "1"
end_id   = "100"

[prompt]
template = "prompts/base.md"

[model]
provider = "langchain_mistralai.ChatMistralAI"
name     = "ministral-8b-2512"

[adapter]
callable = "adapter.soc_adapter"

[solver]
reformat_prompt              = "That answer did not parse. Reply with ONLY 'TRIPLE(x, y, z)'."
tools                        = ["julia", "notebook"]
max_steps                    = 20
solution_adaption_attempts   = 3
context_window_size          = 128_000
context_compaction_threshold = 0.8
error_summarization_threshold = 500

[tools.julia]
callable          = "ape.tools.Julia"
response_timeout  = 135
execution_timeout = 120

[tools.notebook]
callable = "ape.tools.SessionNotebook"

[checker]
callable       = "checker.SOCChecker"
expose_as_tool = true
checker_tool   = "tool.soc_checker_tool"

[run]
save_file      = "output/results.jsonl"
metadata_file  = "output/metadata.json"
solver_workers = 4
checker_workers = 2
runs_per_data  = 1
load_save      = true

[cli]
enabled          = true
log_file         = "output/run.log"
session_log_file = "output/sessions.log"
log_level        = "INFO"

See the TOML configuration reference for the full list of available keys and their defaults.