Building a Custom Benchmark¶

This tutorial walks through building a complete benchmark from scratch. It goes beyond the Quickstart by showing a realistic multi-file layout, a typed solution class, a checker that uses that type, and how to expose the checker as an LLM tool.

The running example is the Sum of Three Cubes problem: given an integer k, find integers
x, y, z such that x³ + y³ + z³ = k. Verification is trivial arithmetic, but finding solutions can require large search spaces — a good fit for APE.

What you need to implement¶

Every APE benchmark requires three mandatory pieces and one that is almost always needed in practice:

The adapter is technically optional only if your checker accepts a plain string — in practice you will always write one.

Prerequisites¶

• APE installed (pip install ape-framework)
• A LangChain chat model provider installed and an API key configured

APE works with any LangChain chat model. Install the package for the provider you want to use and export the corresponding API key:

For example, to use Mistral:

pip install langchain-mistralai
export MISTRAL_API_KEY="your-key"

The [model] section of config.toml tells APE which class and model name to use:

[model]
provider = "langchain_mistralai.ChatMistralAI"
name = "ministral-8b-2512"

Replace provider with the fully qualified class name from the package you installed
(e.g. langchain_openai.ChatOpenAI) and name with the model identifier your provider expects.

1. Create the problem folder¶

APE has no required directory structure, but a consistent layout makes problems easy to navigate. Create a folder for your problem:

sum-of-cubes/

Everything below is placed inside this folder. All paths in config.toml will be relative to it.

2. Define the dataset¶

Create data/schema.py with the schema for one row of your dataset:

from ape.data import DataModel, id_field

class SOCSchema(DataModel):
    k: int = id_field()

DataModel is a Pydantic model. id_field() marks k as the row identifier — APE uses it for logging, saving progress, and resume. You can add as many additional fields as your dataset provides; they are available in the prompt template.

Create data/__init__.py so the package is importable:

from .schema import SOCSchema

Now create data/problems.csv with the problems you want to evaluate:

k
33
42
3

APE supports both .csv and .jsonl as input formats.

3. Write the prompt template¶

Create prompts/base.md. APE renders this as a mustache template once per data item, substituting {{ field_name }} with the value from the schema:

## Task — Sum of Three Cubes

Let k be {{ k }}. Find integers x, y, z such that

    x³ + y³ + z³ = k

## Response format

Output exactly `TRIPLE(x, y, z)` where x, y, z are your integers.
Do not include any explanation.

4. Define the solution type¶

The checker needs a structured object, not a raw string. Create a solution class in checker.py:

class TripleSolution:
    def __init__(self, x: int, y: int, z: int) -> None:
        self.x = x
        self.y = y
        self.z = z

    def __str__(self) -> str:
        return f"({self.x}, {self.y}, {self.z})"

Using a dedicated class (rather than a plain tuple) makes the checker signature self-documenting and lets you add methods later without changing the interface.

5. Write the adapter¶

The adapter is a function that parses the LLM's raw text response into a TripleSolution. Create adapter.py:

import re
from checker import TripleSolution

def soc_adapter(raw: str) -> TripleSolution:
    matches = re.findall(r"TRIPLE\((-?\d+),\s*(-?\d+),\s*(-?\d+)\)", raw)
    if not matches:
        raise ValueError(f"No triple found in response: {raw!r}")
    return TripleSolution(*map(int, matches[-1]))

Key points:

• Raise ValueError (or any exception) when parsing fails. APE catches it and, if configured, shows the LLM a reformat prompt and retries.
• Taking the last match handles responses where the model revises itself before giving a final answer.
• Negative integers must be matched explicitly — note the -? in the regex.

6. Write the checker¶

Complete checker.py by adding the SOCChecker class below TripleSolution. The full file now looks like this:

import time
from datetime import timedelta
from typing import override

from ape.checker import Checker, CheckResult, Failed, Passed
from data import SOCSchema

class TripleSolution:
    def __init__(self, x: int, y: int, z: int) -> None:
        self.x = x
        self.y = y
        self.z = z

    def __str__(self) -> str:
        return f"({self.x}, {self.y}, {self.z})"

class SOCChecker(Checker[SOCSchema, TripleSolution]):

    @override
    def check(self, data: SOCSchema, solution: TripleSolution) -> CheckResult:
        start = time.perf_counter()

        correct = solution.x**3 + solution.y**3 + solution.z**3 == data.k

        runtime = timedelta(seconds=time.perf_counter() - start)
        if not correct:
            return Failed(reason="Incorrect sum of cubes", runtime=runtime)
        return Passed(runtime=runtime)

Checker[DataT, SolutionT] is generic. The type parameters document exactly what the checker expects, which APE uses for validation. Passed and Failed both accept an optional runtime and metadata dict.

7. Wire it all together¶

Create config.toml:

[data]
path = "data/problems.csv"
schema = "data.SOCSchema"

[prompt]
template = "prompts/base.md"

[model]
provider = "langchain_mistralai.ChatMistralAI"
name = "ministral-8b-2512"

[adapter]
callable = "adapter.soc_adapter"

[solver]
reformat_prompt = "That answer did not parse. Reply with ONLY 'TRIPLE(x, y, z)'."

[checker]
callable = "checker.SOCChecker"

APE resolves dotted module paths (data.SOCSchema, adapter.soc_adapter, checker.SOCChecker) relative to the folder containing config.toml. No package installation is needed — just put the files in the same directory.

8. Run the benchmark¶

python -m ape config.toml

APE validates the configuration, starts a live CLI dashboard, and writes results to output/ when done. See Outputs & Results for what each file contains.

9. Expose the checker as an LLM tool (optional)¶

For hard search problems the LLM benefits from being able to verify candidate triples mid-session, before committing to a final answer. APE can expose any checker as a LangChain tool with two config keys and a small factory function.

Create tool.py:

import json

from langchain_core.tools import BaseTool, tool

def soc_checker_tool() -> BaseTool:
    """Build a LangChain tool that checks x³ + y³ + z³ = k."""

    @tool
    def check_triple(x: int, y: int, z: int, k: int) -> str:
        """Check whether (x, y, z) solves x³ + y³ + z³ = k.

        Args:
            x: first integer
            y: second integer
            z: third integer
            k: target integer
        """
        if x**3 + y**3 + z**3 == k:
            return json.dumps({"status": "success", "result": "OK"})
        return json.dumps({
            "status": "success",
            "result": f"Incorrect (got {x**3 + y**3 + z**3}, expected {k})",
        })

    return check_triple

Then add two keys to the [checker] section in config.toml:

[checker]
callable = "checker.SOCChecker"
expose_as_tool = true
checker_tool = "tool.soc_checker_tool"

The factory (soc_checker_tool) is called with the same keyword arguments as the checker constructor, so a single set of [checker] keys configures both. The LLM can now call check_triple between reasoning steps.

Final folder layout¶

sum-of-cubes/
  config.toml
  adapter.py
  checker.py
  tool.py               # only if expose_as_tool = true
  prompts/
    base.md
  data/
    __init__.py
    schema.py
    problems.csv
  output/               # written by APE, gitignore this

Next steps¶

• Add more prompt templates and switch between them with [prompt] template.
• Scale up with parallel workers — see Advanced LLM Run.
• Run on a slice of the dataset with start_id / end_id in [data].
• Attach computation tools (Julia, SessionNotebook) — also in Advanced LLM Run.