Development Setup¶

This page covers setting up a local environment to work on APE itself — running the test suite, the formatting and type-checking gates, and building the docs. If you only want to use the framework, see Installation instead.

Requirements¶

• Python 3.12 or newer. The codebase uses 3.12 syntax (PEP 695 generics such as class Solver[DataT, SolutionT] and type aliases), so older interpreters will not import it.
• Git, for the pre-commit hooks.

Clone and install¶

Work from a virtual environment and install the package in editable mode with the dev extra:

git clone https://github.com/matikasp/Algebra-Problems-Evaluator.git
cd Algebra-Problems-Evaluator

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

pip install -e .[dev]

The dev extra pulls in the toolchain used by CI: pre-commit, black, isort, mypy, plus two LangChain provider packages: langchain-google-genai (required by the integration test in tests/integration_run/, which patches ChatGoogleGenerativeAI) and langchain-mistralai (used only by demos/solver.py, not by ape itself). The optional docs extra (below) is separate.

Code style and type checking¶

APE enforces three gates through pre-commit, pinned in .pre-commit-config.yaml:

Install the git hook once so the gates run on every commit:

pre-commit install

Run them manually across the whole tree at any time:

pre-commit run --all-files

mypy is configured to disallow_untyped_defs, disallow_incomplete_defs, no_implicit_optional, warn_return_any, warn_unused_ignores, and strict_equality over the ape package. New code is expected to be fully annotated.

Running the tests¶

The test suite uses the standard library unittest. Discover and run everything under tests/:

python -m unittest discover -s tests

To run a single module or test case:

python -m unittest tests.test_solver_config
python -m unittest tests.test_workflow.SomeTestCase.test_something

A few notes on the suite:

• Unit tests for the workflow and orchestrators avoid real network calls by driving a scripted, fake chat model (tests/scripted_chat_model.py) instead of a live provider.
• The end-to-end test in tests/integration_run/ exercises the full config-driven path (ape.run) against a tiny bundled benchmark, patching out the LLM provider.
• The external-tool tests (test_julia, test_lmfdb, test_notebook) fully mock their network and database dependencies (urllib.request.urlopen, psycopg's ConnectionPool, etc.), so the entire suite runs offline — no Julia server or LMFDB database needed.

Demo scripts¶

The demos/ directory contains small standalone scripts that exercise individual pieces of the framework against real providers — useful when you want to sanity-check a change end-to-end without setting up a full benchmark. Highlights:

• demos/solver.py — drives a single Solver against a Mistral model (uses langchain-mistralai from the dev extra; needs MISTRAL_API_KEY).
• demos/julia_tool.py, demos/lmfdb_tool.py — invoke the bundled tools directly, useful when you do have the corresponding service running locally.
• demos/run_cli.py — an end-to-end demo of Runner.run + the CLI dashboard driven by a scripted chat model (no network, no API key); run from the repo root with python -m demos.run_cli.
• demos/demo_config_run/demo/ — a self-contained mini-benchmark (schema, adapter/checker components, prompt, and config.toml) you can launch with python -m ape demos/demo_config_run/demo/config.toml to exercise the full TOML-driven path.

For a step-by-step walk-through of writing a benchmark from scratch, see the Quickstart.

Building the documentation¶

The docs are built with MkDocs and the API reference is generated from docstrings by mkdocstrings. Install the docs extra:

pip install -e .[docs]

Serve the site locally with live reload:

mkdocs serve

Build it exactly as CI does — strict turns warnings (such as a broken cross-reference or a page missing from the nav) into errors:

mkdocs build --strict

Because the API pages embed ape.<module> directives, mkdocstrings imports the package at build time. Build from the same environment where APE is installed, or the import will fail.

Continuous integration and delivery¶

Three GitHub Actions workflows guard the project:

• CI (.github/workflows/ci.yml) — runs pre-commit run --all-files and python -m unittest discover -s tests on pushes and PRs to main / dev.
• Docs (.github/workflows/docs.yml) — runs mkdocs build --strict on every push and PR (so a broken cross-reference blocks the PR), and on merges to main / dev publishes versioned docs with mike (main → latest, dev → dev). The deploy job is skipped on PRs (if: github.event_name != 'pull_request').
• CD (.github/workflows/cd.yml) — on every push to main, builds the ape-framework distribution and publishes it to PyPI via pypa/gh-action-pypi-publish. The job first checks that the version in pyproject.toml is strictly greater than the version already on PyPI and fails otherwise, so a release is triggered by bumping the version in pyproject.toml in the merge to main.

Running pre-commit run --all-files, python -m unittest discover -s tests, and mkdocs build --strict locally reproduces the full set of pre-merge checks before you open a pull request.