Data

The data layer defines what one problem instance looks like and how problems are read from disk. Implement a DataModel subclass for your benchmark, then let APE load it through a DataProvider (chosen automatically from the input file extension).

data

DataModel

Bases: BaseModel

Base class for all schemas. Every schema designates one of its fields as the unique identifier -- used across the framework for UI display, save persistence, etc.

By default, the framework reads the identifier from a field named id. Schemas whose natural identifier has a different name should mark it with id_field()::

class IGPSchema(DataModel):
    # Default; id field is the identifier.
    id: str = Field(alias="TransitiveId")
    ...

class CubeProblem(DataModel):
    # k is the identifier.
    k: int = id_field()
    x: int
    y: int
    z: int

The id can be any type with a meaningful __str__ (true of all builtin types) -- the framework coerces to str when needed.

data_id property 

data_id: Any

The value of this item's identifier field. The framework reads it for UI display, save persistence, and resume; it is coerced to str wherever a string id is needed.

CSVDataProvider 

CSVDataProvider(schema: type[DataT], input_path: Path, start_id: str | None = None, end_id: str | None = None)

Bases: DataProvider[DataT]

DataProvider for .csv files. Each row becomes one record and empty cells are read as None.

Open input_path and prepare to yield records validated as schema.

Parameters:
  • schema (type[DataT]) –

    The DataModel subclass each record is validated against.

  • input_path (Path) –

    Path to the input file.

  • start_id (str | None, default: None ) –

    If given, skip records until the item with this id (the window is inclusive of it).

  • end_id (str | None, default: None ) –

    If given, stop after the item with this id (inclusive).

DataProvider 

DataProvider(schema: type[DataT], input_path: Path, start_id: str | None = None, end_id: str | None = None)

Bases: ABC, Iterator[DataT]

Reads problem instances from a file and yields them as validated DataModel objects of schema DataT. Iterate over a provider to consume items one at a time; the concrete subclass (chosen by load) handles the file format.

Construct one with DataProvider.load rather than instantiating a subclass directly.

Open input_path and prepare to yield records validated as schema.

Parameters:
  • schema (type[DataT]) –

    The DataModel subclass each record is validated against.

  • input_path (Path) –

    Path to the input file.

  • start_id (str | None, default: None ) –

    If given, skip records until the item with this id (the window is inclusive of it).

  • end_id (str | None, default: None ) –

    If given, stop after the item with this id (inclusive).

count 

count() -> int

Returns the total amount of records in the file, respecting start_id/end_id.

load classmethod 

load(input_path: Path, schema: type[DataT], start_id: str | None = None, end_id: str | None = None) -> DataProvider[DataT]

Build the right provider for input_path based on its extension.

Parameters:
  • input_path (Path) –

    Input file; .csv and .jsonl are supported.

  • schema (type[DataT]) –

    The DataModel subclass to validate records against.

  • start_id (str | None, default: None ) –

    Optional inclusive lower bound on the item id to start from.

  • end_id (str | None, default: None ) –

    Optional inclusive upper bound on the item id to stop at.
Returns:
  • DataProvider[DataT] –

    A CSVDataProvider or JSONLDataProvider for the file.
Raises:
  • ValueError –

    If the file extension has no associated provider.

JSONLDataProvider 

JSONLDataProvider(schema: type[DataT], input_path: Path, start_id: str | None = None, end_id: str | None = None)

Bases: DataProvider[DataT]

DataProvider for .jsonl files. Each non-empty line is parsed as one JSON record.

Open input_path and prepare to yield records validated as schema.

Parameters:
  • schema (type[DataT]) –

    The DataModel subclass each record is validated against.

  • input_path (Path) –

    Path to the input file.

  • start_id (str | None, default: None ) –

    If given, skip records until the item with this id (the window is inclusive of it).

  • end_id (str | None, default: None ) –

    If given, stop after the item with this id (inclusive).

id_field 

id_field(**kwargs: Any) -> Any

Drop-in replacement for pydantic.Field() that also marks the field as the schema's unique identifier. Accepts all the same kwargs as Field().

Each schema must designate exactly one field as the identifier -- either by naming it id (no marker required) or by using id_field() on a field with a different name.

Example::

class CubeProblem(DataModel):
    k: int = id_field()  # k is the identifier
    x: int
    y: int
    z: int

class IGPSchema(DataModel):
    transitive_id: str = id_field(alias="TransitiveId")
    # ...