Checker

The checker is the deterministic verifier of the pipeline. Implement Checker.check to return Passed or Failed for a (data, solution) pair. The CheckerOrchestrator runs a pool of checker instances concurrently and is managed by the runner — you normally only implement Checker and hand the framework a CheckerFactory.

checker

CheckerFactory 

CheckerFactory = Callable[[], Checker[DataT, SolutionT]]

Zero-argument callable that returns a fresh Checker. Called once per checker worker so each worker gets its own instance; a Checker subclass whose constructor takes no arguments is itself a valid factory.

CheckResult 

CheckResult = Passed | Failed

The outcome of a single check: either Passed or Failed.

Checker

Bases: ABC

Deterministic verifier of the pipeline. Subclass it and implement check for your benchmark; the framework runs a pool of instances concurrently via a CheckerFactory and a CheckerOrchestrator.

The type parameters are the data-item schema (DataT, a DataModel subclass) and the parsed solution type produced by your SolutionAdapter.

check abstractmethod 

check(data: DataT, solution: SolutionT) -> CheckResult

Verify solution against the problem instance data.

Parameters:
  • data (DataT) –

    The problem instance the solution was produced for.

  • solution (SolutionT) –

    The parsed solution to verify.
Returns:

Failed dataclass 

Failed(reason: str, runtime: Optional[timedelta] = None, metadata: dict[str, Any] = dict())

Outcome of a check the solution did not satisfy. Return this from Checker.check when the solution is wrong.

reason instance-attribute 

reason: str

Short, human-readable explanation of the failure. The visualizer groups attempts into failure classes by this string, so keep it stable and low-cardinality (a category, not a per-attempt message).

runtime class-attribute instance-attribute 

runtime: Optional[timedelta] = None

Wall-clock time the check took, if measured. Surfaced in the run summary.

metadata class-attribute instance-attribute 

metadata: dict[str, Any] = field(default_factory=dict)

Arbitrary extra data to attach to the result; persisted with the output.

Passed dataclass 

Passed(runtime: Optional[timedelta] = None, metadata: dict[str, Any] = dict())

Outcome of a check the solution satisfied. Return this from Checker.check when the solution is correct.

runtime class-attribute instance-attribute 

runtime: Optional[timedelta] = None

Wall-clock time the check took, if measured. Surfaced in the run summary.

metadata class-attribute instance-attribute 

metadata: dict[str, Any] = field(default_factory=dict)

Arbitrary extra data to attach to the result; persisted with the output.

CheckerOrchestrator 

CheckerOrchestrator(checker_factory: CheckerFactory[DataT, SolutionT], checker_workers: int, tracker: WorkerStatusTracker, retry_timeout: int = 10)

Runs a fixed pool of Checker instances concurrently on behalf of the runner. You normally do not construct this yourself — the Runner builds it from your CheckerFactory — but it is the bridge between solver workers and your checkers.

A thread pool of checker_workers threads is created, each leasing one Checker from a carousel, so a given checker instance only ever runs one check at a time.

Build the pool.

Parameters:
  • checker_factory (CheckerFactory[DataT, SolutionT]) –

    Called checker_workers times to build one Checker per worker.

  • checker_workers (int) –

    Number of checker threads (and checker instances).

  • tracker (WorkerStatusTracker) –

    Worker-status tracker updated as checks start and finish.

  • retry_timeout (int, default: 10 ) –

    Seconds between cancellation checks while waiting for a check to complete.

check 

check(data: DataT, solution: SolutionT, solver_id: int) -> CheckResult

Check one (data, solution) pair on the pool and return the result.

Blocks until a checker worker produces a result, polling on an internal timeout so the call can be interrupted when the run is shutting down.

Parameters:
  • data (DataT) –

    The problem instance.

  • solution (SolutionT) –

    The parsed solution to verify.

  • solver_id (int) –

    Id of the solver worker that produced the solution, used for dashboard status reporting.
Returns:
  • CheckResult –

    The CheckResult from the underlying Checker.
Raises:
  • InterruptedError –

    If the orchestrator has been stopped.

stop 

stop() -> None

Signal shutdown and cancel pending checks. Afterwards check raises InterruptedError and in-flight futures are cancelled.