Quickstart¶

Build a tiny Sum of Squares benchmark from scratch and run it with the CLI. This guide assumes APE is already installed.

1. Install a model provider¶

Pick any LangChain chat model provider. This example uses Mistral.

python -m pip install langchain-mistralai
export MISTRAL_API_KEY="your-key"

2. Create the dataset¶

Create data.csv:

3. Create the schema¶

Create data.py:

from ape.data import DataModel, id_field

class SquareProblem(DataModel):
    k: int = id_field()

4. Create the prompt¶

Create prompt.md:

## Task - Sum of Two Squares
Let k be {{ k }}. Generate an integer x, y that solves

    x^2 + y^2 = k

## Response Format
Output exactly two integers separated by spaces: "x y". 
No commas, no labels, and no explanation.

5. Create the adapter¶

Create adapter.py:

def parse_answer(raw: str) -> tuple[int, int]:
    line = next((ln for ln in raw.splitlines()[::-1] if ln.strip()), "")
    parts = line.split()
    if len(parts) != 2:
        raise ValueError(f"Expected two space-separated integers, got: {raw!r}")
    return tuple(map(int, parts))

6. Create the checker¶

Create checker.py:

import time
from datetime import timedelta
from typing import override

from ape.checker import Checker, CheckResult, Failed, Passed
from data import SquareProblem

class SquareChecker(Checker[SquareProblem, tuple[int, int]]):
    @override
    def check(self, data: SquareProblem, solution: tuple[int, int]) -> CheckResult:
        start = time.perf_counter()
        x, y = solution
        correct = x**2 + y**2 == data.k
        runtime = timedelta(seconds=time.perf_counter() - start)
        if not correct:
            return Failed(reason="Incorrect sum of squares", runtime=runtime)
        return Passed(runtime=runtime)

7. Create the config¶

Create config.toml:

[data]
path = "data.csv"
schema = "data.SquareProblem"

[prompt]
template = "prompt.md"

[model]
provider = "langchain_mistralai.ChatMistralAI"
name = "ministral-8b-2512"

[adapter]
callable = "adapter.parse_answer"

[solver]
reformat_prompt = "That answer did not parse. Reply with ONLY the final answer."

[checker]
callable = "checker.SquareChecker"

If you use another provider, replace the [model] provider and name, and install its package (for example, langchain-openai, langchain-google-genai, etc.).

8. Run the benchmark¶

python -m ape config.toml

You will see a live CLI dashboard while the run is in progress.

9. Inspect results¶

The run writes output files into an output/ directory next to the config:

results.jsonl: per-item results
metadata.json: run metadata and solver configuration
run.log: CLI logs
sessions.log: raw LLM transcripts in human-readable format
sessions.jsonl: raw LLM transcripts in JSON Lines format

10. Visualize¶

The run automatically creates a summary chart in the output folder. If you want to re-generate it later, run:

python -m ape visualize output/