LLM Backend Design¶

This page describes the internals of the solver — the part of the framework that turns a rendered prompt into a parsable solution. The LLM Integration user-guide page covers the same loop from a configuration point of view (which knobs exist and what they do); this page covers the code structure and the reasoning behind it, for people changing it.

The backend is built on LangChain (model abstraction and tools) and LangGraph (the solve loop as an explicit state machine).

Three layers¶

The solver is deliberately split into three objects with shrinking scope:

Solver.from_config constructs a solver from the config; solver.solve(session, data) builds a fresh Workflow, compiles it, and streams it to completion. A new workflow per solve keeps each attempt's state fully isolated — important because solver workers run concurrently and runs_per_data may launch several attempts on the same item.

The graph¶

The workflow is a StateGraph over a small typed State:

class State(MessagesState):
    performed_steps: int
    result: Result | None

MessagesState provides the running messages list; performed_steps counts "think" turns against max_steps; result holds the final Result once a terminal node sets it.

The nodes and the transitions between them:

        START
          │
          ▼
        think ◄───────────────────────────────────────────────┐
          │                                                   │
          ▼                                                   │
        route ── (tool calls, context OK) ─────────────► use_tools
          │  │                                                ▲
          │  └ (tool calls, context full) ─► compact_context ─┘  (straight to tools)
          │                                                    
          │                                        
          ├── (plain answer) ─────────────────► adapt_solution ──► END
          │                                            ▲
          └── (performed_steps > max) ─► force_finish ─┘

think and the edge think → route are fixed; everything else is decided at runtime by route, which returns a LangGraph Command(goto=...) rather than relying on static conditional edges. Centralizing the branching in one node keeps the policy ("when do we use tools / compact / finish?") in a single readable place.

Node responsibilities¶

• think — calls the model on the current history, appends the reply, and increments performed_steps.
• route — the policy node. In order: if the step budget is exceeded, resolve any dangling tool calls and go to force_finish; if the last message has no tool calls, go to adapt_solution; if context occupancy is at/over the threshold, go to compact_context; otherwise go to use_tools.
• use_tools — a LangGraph ToolNode that executes the pending tool calls. Its outputs are post-processed before re-entering think (see Tool output formatting).
• compact_context — summarizes the middle of the conversation and rewrites the message list (see Context compaction).
• force_finish — the model has spent its step budget; ask it for a final answer with tool calls disabled, then fall through to adapt_solution. Both the injected feedback HumanMessage and the model's reply are returned in the node update so the solver can record them as separate entries in the session log.
• adapt_solution — runs the SolutionAdapter on the last message, retrying with the reformat prompt on failure; sets result.

Prompts¶

The framework-injected prompts are plain Markdown files shipped with the package under ape/llm/prompts/ (system.md, compact-context.md, force-finish.md, summarize-error.md). Their lifetimes differ:

• system.md is read once in Solver.__init__ and cached on the solver (joined with the benchmark's user_system_prompt if any — the base is always prepended, never replaced).
• compact-context.md, force-finish.md, summarize-error.md are read in Workflow.__init__, i.e. afresh on every solve() call.

Resolving the model: _invoke_model¶

Every model call across all nodes goes through Workflow._invoke_model, the one place that owns retry behaviour. It walks rate_limit_backoff_seconds, sleeping the listed number of seconds between attempts; the length of that tuple is the number of attempts, so the schedule and the retry count are configured by a single value. If every attempt fails it raises LLMInvokeErr, which the solver turns into an INTERNAL_ERROR result. The cancellation Event is checked between attempts — before each invoke and again before each backoff sleep — but the time.sleep(backoff) itself is uninterruptable, so a shutdown that arrives mid-backoff will still wait out the remaining sleep (up to max(rate_limit_backoff_seconds) seconds, 32 s by default). Keep that in mind if you raise the backoff schedule.

A subtlety that recurs: several nodes invoke the model with self._llm.bind(tool_choice="none"). force_finish, adapt_solution, and compact_context all want plain text, and some providers reject a message ordering or demand a tool response when tools are bound — forbidding tool calls for those turns avoids provider-specific 400s.

Tools¶

Tools are bound once, right after construction, in Workflow.add_tools: the unbound BaseChatModel becomes a tools-bound Runnable via bind_tools, and a ToolNode is added as the use_tools node. The model field _llm is typed as a Runnable precisely because it starts as a chat model and may become a bound runnable.

Tool output formatting¶

use_tools wraps each tool call with _format_tool_output, which does two things to the raw ToolMessage:

1. Normalizes the JSON the tool returns into a compact, model-friendly STATUS: … block (the built-in tools return a dict with a status key plus payload fields).
2. Summarizes long errors. If the tool reports an error whose text is at least error_summarization_threshold characters, the workflow asks an LLM to summarize it — using the helper model if one is configured, falling back to the main model — and records that it summarized (and the original length) in the message's additional_kwargs. If the helper itself fails, or returns something no shorter than the original, it falls back to plain truncation. The principle is best-effort: a flaky helper or a giant stack trace must never kill the run, only degrade the message.

Context compaction¶

The workflow tracks how full the context window is and compacts rather than truncates. Occupancy is estimated by _context_window_occupation with the o200k_base tokenizer (a provider-neutral approximation, since BaseChatModel.get_num_tokens is unreliable across providers), counting both the message history and the JSON tool schemas. WorkflowParams resolves the window size from the model's LangChain profile when it isn't set explicitly, defaulting to 256k tokens.

When route sees occupancy at/above context_compaction_threshold with tool calls pending, compact_context asks the LLM (with tool_choice="none") to summarize messages[1:-1] — the original instruction plus the middle of the conversation, dropping only the system prompt (index 0) and the last message (the pending tool-call AIMessage). The summary then replaces the middle, so the rebuilt list is [system, instruction, summary, last]: the instruction both stays verbatim and informs the summary's content. It then routes straight to use_tools with an Overwrite of the messages, deliberately bypassing route: compaction is only ever reached with tool calls pending, so going back through route could re-check occupancy and loop without advancing performed_steps if the threshold were set very low.

Terminating: force_finish and adapt_solution¶

There are two ways to leave the loop, and both converge on adapt_solution:

• The model stops calling tools on its own → route sends it to adapt_solution.
• The model exhausts max_steps → force_finish injects a "give me your final answer now" prompt (tools disabled) and edges into adapt_solution.

adapt_solution then loops up to solution_adaption_attempts times: it feeds the last message to the adapter, and on a parse exception it appends the reformat_prompt and re-invokes the model (again with tool_choice="none", so the retry is always plain text). Two early exits short-circuit this: if the model output contains the literal I_GIVE_UP the result is Err(GAVE_UP); if all attempts fail to parse it is Err(PARSING_FAILED). Otherwise it is Ok(solution).

The Session: how the solver talks to the rest of the framework¶

The solver does not write to the dashboard or the logs directly. Each solve() is handed a Session — created fresh by the orchestrator — that is the only channel out:

• Status — session.report(state, session_length) pushes a SolverState (SOLVING, AWAITING_TOOL, AWAITING_CHECK, IDLE) to the worker dashboard.
• Transcript — session.add(kind, turn, fields) buffers one EntryKind entry. The full set is PROMPT, LLM_RESPONSE, TOOL_CALL, TOOL_OUTPUT, FEEDBACK, COMPACT_CONTEXT, CHECKER_RESULTS, FAILURE, SESSION_END. session.close(reason, turn) appends the terminal SESSION_END entry and flushes the whole transcript exactly once at the end.
• Header summary — session.data_summary is a one-line string written into the session-log header for each session. It defaults to str(data.data_id), but solvers are encouraged to overwrite it with something more descriptive (the built-in Solver uses a short rendering of the data item) before calling close.
• Cancellation — session.cancelled() exposes the SolverOrchestrator's cancellation Event (shared across every session it spawns, but distinct from the CheckerOrchestrator's own event).

Solver.solve consumes the compiled graph with compiled.stream(..., stream_mode="updates") and translates each node update into the appropriate session entries — this is where the workflow's internal steps become the human-readable sessions.log and the machine-readable sessions.jsonl (see Outputs & Results). Keeping logging in the streaming loop rather than inside the nodes means the workflow stays a pure state machine and the session stays the single sink for everything observable about a solve.

Error taxonomy¶

The internal failure modes map onto the ErrKind values the solver returns, and through OutputBatch onto the labels you see in the visualizer:

Two internal exceptions support this: LLMInvokeErr (retries exhausted) and CorruptedContextErr (the message history violated an invariant a node relies on, e.g. the last message wasn't an AIMessage); both surface as INTERNAL_ERROR.