LLM Integration¶

APE uses LangChain as its model abstraction layer. Any BaseChatModel implementation works as the solver backend.

Choosing a provider¶

The [model] config section picks the provider and model name:

[model]
provider = "langchain_mistralai.ChatMistralAI"
name = "ministral-8b-2512"

provider is a fully-qualified dotted path to a LangChain chat model class. name becomes its model= constructor argument. Install the matching LangChain integration package and set the required API key before running:

APE loads a .env file automatically if one exists in the project root or any of its parent directories, so you can keep keys out of shell history.

The solve workflow¶

Each solver session runs a LangGraph workflow. The graph drives the model through a loop until it produces a parsable answer or hits a limit:

START
  │
  ▼
think  ◄──────────────────────────────────────────────────┐
  │                                                       │
  ▼                                                       │
route                                                     │
  ├─ tool calls pending ─────────────────────► use_tools ─┘
  │     │                                           ▲
  │     └─► context near limit ──► compact_context ─┘
  │
  ├─ no tool calls ───────────────► adapt_solution ──► END
  │                                      ▲
  └─ max_steps reached ──► force_finish ─┘

think — invokes the model with the current message history and appends its reply.

route — decides what comes next. The conditions are checked in this order, so earlier ones take precedence:

• If max_steps has been reached, go to force_finish (any pending tool calls are resolved first so the model provider stays happy).
• Otherwise, if the model produced a plain text reply (no tool calls), go to adapt_solution.
• Otherwise (tool calls pending), if the context is at or above the compaction threshold, go to compact_context first.
• Otherwise, go to use_tools.

use_tools — executes all pending tool calls and appends their outputs.

force_finish — sends a prompt asking the model to produce a final answer immediately (tool calls are disabled for this turn). The session then proceeds to adapt_solution, which parses the forced answer like any other, so the outcome can be Ok(solution), PARSING_FAILED, or GAVE_UP (if the forced answer still contains I_GIVE_UP).

adapt_solution — passes the last model message to the SolutionAdapter. On failure it sends the reformat_prompt and retries, up to solution_adaption_attempts total attempts. If the model output contains the literal string I_GIVE_UP at any attempt, the session ends with GAVE_UP immediately.

Context compaction¶

When the conversation grows long, APE compacts it automatically rather than truncating or failing. The workflow estimates context occupancy using the o200k_base (GPT-4o/GPT-5) tokenizer as a provider-neutral approximation. Token counts include both message history and tool schemas.

When occupancy reaches context_compaction_threshold (default 0.8), APE summarizes the history into a single message and continues the session:

[solver]
context_window_size = 131072          # set manually if auto-detection fails
context_compaction_threshold = 0.75   # trigger earlier

If context_window_size is not set, APE tries to read it from the model's LangChain profile and falls back to 256 000 tokens.

Helper LLM¶

A second, cheaper model can handle tasks that do not require the full solver: context compaction and long tool-error summarization. Set it in the [model] section:

[model]
provider = "langchain_openai.ChatOpenAI"
name = "gpt-4.1"

helper_provider = "langchain_openai.ChatOpenAI"
helper_name = "gpt-4.1-mini"

If no helper is configured, the main model handles these tasks too.

System prompt¶

Every session starts with a built-in system prompt that sets general expectations for the solver. You can extend it without replacing it:

[solver]
user_system_prompt = "prompts/system_extension.md"

The value can be either a file path (relative to the config) or an inline string. The user prompt is appended to the built-in one.

Rate limits and retries¶

LLM calls are retried automatically when they fail. The rate_limit_backoff_seconds tuple sets the wait time before each attempt; its length is also the total number of attempts per call:

[solver]
rate_limit_backoff_seconds = [2, 4, 8, 16, 32]   # 5 attempts, default

If all attempts fail, the session ends with INTERNAL_ERROR.

Key solver parameters¶