Runner¶

Entry point that drives the whole evaluation pipeline. Its single classmethod run wires together the solver and checker orchestrators, the save handler, and the worker-status tracker, then streams results to the optional RunObserver (e.g. the CLI) until every data item is done.

There is no instance state — call Runner.run(...) directly.

run(solver_config: SolverConfig[SolutionT], checker_params: dict[str, Any], checker_factory: CheckerFactory[DataT, SolutionT], data_provider: DataProvider[DataT], params: Optional[RunParams] = None, observer: Optional[RunObserver] = None, on_batch_saved: Optional[Callable[[OutputBatch[DataT, SolutionT]], None]] = None) -> None

Bases: StrEnum

States a checker worker passes through. Set by the CheckerOrchestrator around each checker.check call.

A checker worker's last-reported status, as shown on the dashboard.

id: int

state: CheckerState

data_id: Optional[Any]

solver_id: Optional[int]

Default observer used when a caller doesn't pass one — drops every event on the floor. Lets Runner and WorkerStatusTracker treat "no observer" identically to "an observer that ignores everything", so internal state always stays consistent.

Bases: Protocol

The single channel through which a run reports progress to the outside world. Implement this on whatever consumes run telemetry — e.g. CLI does so to drive its dashboard. Pass an instance to Runner.run(observer=...).

Both methods are called from the runner's main thread (on_progress) or from the tracker's pump thread (on_worker_status). Implementations must be thread-safe with respect to themselves but do not need locking against each other.

on_progress(done: int, total: int) -> None

on_worker_status(snapshot: WorkerSnapshot) -> None

Bases: StrEnum

States a solver worker reports through its Session.

Emitted by the solver itself via session.report(state, ...); the orchestrator never sets these. Add new variants here when the real solver needs to expose more granular phases.

A solver worker's last-reported status, as shown on the dashboard.

id: int

state: SolverState

data_id: Optional[Any]

curr_run: int

total_runs: int

session_length: int

Immutable snapshot of every worker's last-reported status. Pumped to the RunObserver by WorkerStatusTracker.

All attempts the framework made on a single data item. One batch is produced per data item and persisted by the SaveHandler; it pairs each solver result with the checker's verdict for that attempt.

Supports len(batch) (the number of attempts) and iteration, which yields (solver_result, check_result) tuples — check_result is None when the solver failed and the checker was never run.

Create an empty batch for one data item.

get_data() -> DataT

append(solver_result: Result[SolutionT], check_result: Optional[CheckResult]) -> None

pickle() -> str

from_pickle(pickle: str) -> OutputBatch

Parameters that can be passed to the Runner class methods.

save_file: Path = Path('output/results.jsonl')

metadata_file: Path = Path('output/metadata.json')

checker_workers: int = 1

solver_workers: int = 1

runs_per_data: int = 1

load_save: bool = True