Types

Versioning Engine

metaxy.versioning.types.LazyIncrement dataclass

LazyIncrement(*, added: LazyFrame[Any], changed: LazyFrame[Any], removed: LazyFrame[Any], input: LazyFrame[Any] | None = None)

Result of an incremental update containing lazy dataframes.

Attributes:

• added (LazyFrame[Any]) –

  New samples from upstream not present in current metadata.

• changed (LazyFrame[Any]) –

  Samples with different provenance.

• removed (LazyFrame[Any]) –

  Samples in current metadata but not in upstream state.

• input (LazyFrame[Any] | None) –

  Joined upstream metadata with FeatureDep rules applied.

Functions

metaxy.versioning.types.LazyIncrement.collect

collect(**kwargs: Any) -> Increment

Collect all lazy frames to eager DataFrames.

Tip

If all lazy frames are Polars frames, leverages polars.collect_all to optimize the collection process and take advantage of common subplan elimination.

Parameters:

• **kwargs (Any, default: {} ) –

  backend-specific keyword arguments to pass to the collect method of the lazy frames.

Returns:

• Increment ( Increment ) –

  The collected increment.

Source code in src/metaxy/versioning/types.py

def collect(self, **kwargs: Any) -> Increment:
    """Collect all lazy frames to eager DataFrames.

    !!! tip
        If all lazy frames are Polars frames, leverages
        [`polars.collect_all`](https://docs.pola.rs/api/python/stable/reference/api/polars.collect_all.html)
        to optimize the collection process and take advantage of common subplan elimination.

    Args:
        **kwargs: backend-specific keyword arguments to pass to the collect method of the lazy frames.

    Returns:
        Increment: The collected increment.
    """
    if (
        self.added.implementation
        == self.changed.implementation
        == self.removed.implementation
        == nw.Implementation.POLARS
    ):
        polars_eager_increment = PolarsLazyIncrement(
            added=self.added.to_native(),
            changed=self.changed.to_native(),
            removed=self.removed.to_native(),
        ).collect(**kwargs)
        return Increment(
            added=nw.from_native(polars_eager_increment.added),
            changed=nw.from_native(polars_eager_increment.changed),
            removed=nw.from_native(polars_eager_increment.removed),
        )
    else:
        return Increment(
            added=self.added.collect(**kwargs),
            changed=self.changed.collect(**kwargs),
            removed=self.removed.collect(**kwargs),
        )

metaxy.versioning.types.LazyIncrement.to_polars

to_polars() -> PolarsLazyIncrement

Convert to Polars.

Tip

If the Narwhals lazy frames are already backed by Polars, this is a no-op.

Warning

If the Narwhals lazy frames are not backed by Polars, this will trigger a full materialization for them.

Source code in src/metaxy/versioning/types.py

def to_polars(self) -> PolarsLazyIncrement:
    """Convert to Polars.

    !!! tip
        If the Narwhals lazy frames are already backed by Polars, this is a no-op.

    !!! warning
        If the Narwhals lazy frames are **not** backed by Polars, this will
        trigger a full materialization for them.
    """
    return PolarsLazyIncrement(
        added=lazy_frame_to_polars(self.added),
        changed=lazy_frame_to_polars(self.changed),
        removed=lazy_frame_to_polars(self.removed),
        input=lazy_frame_to_polars(self.input) if self.input is not None else None,
    )

metaxy.versioning.types.Increment

Bases: NamedTuple

Result of an incremental update containing eager dataframes.

Contains three sets of samples:

• added: New samples from upstream not present in current metadata

• changed: Samples with different provenance

• removed: Samples in current metadata but not in upstream state

Functions

metaxy.versioning.types.Increment.collect

collect() -> Increment

Convenience method that's a no-op.

Source code in src/metaxy/versioning/types.py

def collect(self) -> "Increment":
    """Convenience method that's a no-op."""
    return self

metaxy.versioning.types.Increment.to_polars

to_polars() -> PolarsIncrement

Convert to Polars.

Source code in src/metaxy/versioning/types.py

def to_polars(self) -> PolarsIncrement:
    """Convert to Polars."""
    return PolarsIncrement(
        added=self.added.to_polars(),
        changed=self.changed.to_polars(),
        removed=self.removed.to_polars(),
    )

metaxy.versioning.types.PolarsIncrement

Bases: NamedTuple

Like Increment, but converted to Polars frames.

Attributes

metaxy.versioning.types.PolarsIncrement.added instance-attribute

added: DataFrame

metaxy.versioning.types.PolarsIncrement.changed instance-attribute

changed: DataFrame

metaxy.versioning.types.PolarsIncrement.removed instance-attribute

removed: DataFrame

metaxy.versioning.types.PolarsLazyIncrement dataclass

PolarsLazyIncrement(*, added: LazyFrame, changed: LazyFrame, removed: LazyFrame, input: LazyFrame | None = None)

Like LazyIncrement, but converted to Polars lazy frames.

Attributes:

• added (LazyFrame) –

  New samples from upstream not present in current metadata.

• changed (LazyFrame) –

  Samples with different provenance.

• removed (LazyFrame) –

  Samples in current metadata but not in upstream state.

• input (LazyFrame | None) –

  Joined upstream metadata with FeatureDep rules applied.

Attributes

metaxy.versioning.types.PolarsLazyIncrement.added instance-attribute

added: LazyFrame

metaxy.versioning.types.PolarsLazyIncrement.changed instance-attribute

changed: LazyFrame

metaxy.versioning.types.PolarsLazyIncrement.removed instance-attribute

removed: LazyFrame

metaxy.versioning.types.PolarsLazyIncrement.input class-attribute instance-attribute

input: LazyFrame | None = None

Functions

metaxy.versioning.types.PolarsLazyIncrement.collect

collect(**kwargs: Any) -> PolarsIncrement

Collect into a PolarsIncrement.

Tip

Leverages polars.collect_all to optimize the collection process and take advantage of common subplan elimination.

Parameters:

• **kwargs (Any, default: {} ) –

  backend-specific keyword arguments to pass to the collect method of the lazy frames.

Returns:

• PolarsIncrement ( PolarsIncrement ) –

  The collected increment.

Source code in src/metaxy/versioning/types.py

def collect(self, **kwargs: Any) -> PolarsIncrement:
    """Collect into a [`PolarsIncrement`][metaxy.versioning.types.PolarsIncrement].

    !!! tip
        Leverages [`polars.collect_all`](https://docs.pola.rs/api/python/stable/reference/api/polars.collect_all.html)
        to optimize the collection process and take advantage of common subplan elimination.

    Args:
        **kwargs: backend-specific keyword arguments to pass to the collect method of the lazy frames.

    Returns:
        PolarsIncrement: The collected increment.
    """
    added, changed, removed = pl.collect_all(
        [self.added, self.changed, self.removed], **kwargs
    )
    return PolarsIncrement(added, changed, removed)

metaxy.HashAlgorithm

Bases: Enum

Supported hash algorithms for field provenance calculation.

These algorithms are chosen for: - Speed (non-cryptographic hashes preferred) - Cross-database availability - Good collision resistance for field provenance calculation

Attributes

metaxy.HashAlgorithm.XXHASH64 class-attribute instance-attribute

XXHASH64 = 'xxhash64'

metaxy.HashAlgorithm.XXHASH32 class-attribute instance-attribute

XXHASH32 = 'xxhash32'

metaxy.HashAlgorithm.WYHASH class-attribute instance-attribute

WYHASH = 'wyhash'

metaxy.HashAlgorithm.SHA256 class-attribute instance-attribute

SHA256 = 'sha256'

metaxy.HashAlgorithm.MD5 class-attribute instance-attribute

MD5 = 'md5'

metaxy.HashAlgorithm.FARMHASH class-attribute instance-attribute

FARMHASH = 'farmhash'

Keys

Types for working with feature and field keys.

Canonical Keys

metaxy.FeatureKey

FeatureKey(parts: str)

FeatureKey(parts: Sequence[str])

FeatureKey(parts: FeatureKey)

FeatureKey(parts: str | Sequence[str] | FeatureKey)

Bases: _Key

Feature key as a sequence of string parts.

Hashable for use as dict keys in registries. Parts cannot contain forward slashes (/) or double underscores (__).

Example:

```py
FeatureKey("a/b/c")  # String format
# FeatureKey(parts=['a', 'b', 'c'])

FeatureKey(["a", "b", "c"])  # List format
# FeatureKey(parts=['a', 'b', 'c'])

FeatureKey(FeatureKey(["a", "b", "c"]))  # FeatureKey copy
# FeatureKey(parts=['a', 'b', 'c'])
```

Source code in src/metaxy/models/types.py

def __init__(
    self,
    parts: str | Sequence[str] | FeatureKey,
) -> None: ...

Attributes

metaxy.FeatureKey.parts property

parts: tuple[str, ...]

Backward compatibility property for accessing root as parts.

metaxy.FeatureKey.table_name property

table_name: str

Get SQL-like table name for this feature key.

Replaces hyphens with underscores for SQL compatibility.

Functions

metaxy.FeatureKey.to_string

to_string() -> str

Convert to string representation with "/" separator.

Source code in src/metaxy/models/types.py

def to_string(self) -> str:
    """Convert to string representation with "/" separator."""
    return KEY_SEPARATOR.join(self.parts)

metaxy.FeatureKey.to_struct_key

to_struct_key() -> str

Convert to a name that can be used as struct key in databases

Source code in src/metaxy/models/types.py

def to_struct_key(self) -> str:
    """Convert to a name that can be used as struct key in databases"""
    return "_".join(self.parts)

metaxy.FeatureKey.__repr__

__repr__() -> str

Return string representation.

Source code in src/metaxy/models/types.py

def __repr__(self) -> str:
    """Return string representation."""
    return self.to_string()

metaxy.FeatureKey.__str__

__str__() -> str

Return string representation.

Source code in src/metaxy/models/types.py

def __str__(self) -> str:
    """Return string representation."""
    return self.to_string()

metaxy.FeatureKey.__lt__

__lt__(other: Any) -> bool

Less than comparison for sorting.

Source code in src/metaxy/models/types.py

def __lt__(self, other: Any) -> bool:
    """Less than comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts < other.parts
    return NotImplemented

metaxy.FeatureKey.__le__

__le__(other: Any) -> bool

Less than or equal comparison for sorting.

Source code in src/metaxy/models/types.py

def __le__(self, other: Any) -> bool:
    """Less than or equal comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts <= other.parts
    return NotImplemented

metaxy.FeatureKey.__gt__

__gt__(other: Any) -> bool

Greater than comparison for sorting.

Source code in src/metaxy/models/types.py

def __gt__(self, other: Any) -> bool:
    """Greater than comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts > other.parts
    return NotImplemented

metaxy.FeatureKey.__ge__

__ge__(other: Any) -> bool

Greater than or equal comparison for sorting.

Source code in src/metaxy/models/types.py

def __ge__(self, other: Any) -> bool:
    """Greater than or equal comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts >= other.parts
    return NotImplemented

metaxy.FeatureKey.__iter__

__iter__() -> Iterator[str]

Return iterator over parts.

Source code in src/metaxy/models/types.py

def __iter__(self) -> Iterator[str]:  # ty: ignore[invalid-method-override]
    """Return iterator over parts."""
    return iter(self.parts)

metaxy.FeatureKey.__getitem__

__getitem__(index: int) -> str

Get part by index.

Source code in src/metaxy/models/types.py

def __getitem__(self, index: int) -> str:
    """Get part by index."""
    return self.parts[index]

metaxy.FeatureKey.__len__

__len__() -> int

Get number of parts.

Source code in src/metaxy/models/types.py

def __len__(self) -> int:
    """Get number of parts."""
    return len(self.parts)

metaxy.FeatureKey.__contains__

__contains__(item: str) -> bool

Check if part is in key.

Source code in src/metaxy/models/types.py

def __contains__(self, item: str) -> bool:
    """Check if part is in key."""
    return item in self.parts

metaxy.FeatureKey.__reversed__

__reversed__()

Return reversed iterator over parts.

Source code in src/metaxy/models/types.py

def __reversed__(self):
    """Return reversed iterator over parts."""
    return reversed(self.parts)

metaxy.FeatureKey.model_dump

model_dump(**kwargs: Any) -> Any

Serialize to string format for JSON dict key compatibility.

Source code in src/metaxy/models/types.py

def model_dump(self, **kwargs: Any) -> Any:
    """Serialize to string format for JSON dict key compatibility."""
    return self.to_string()

metaxy.FeatureKey.__hash__

__hash__() -> int

Return hash for use as dict keys.

Source code in src/metaxy/models/types.py

def __hash__(self) -> int:
    """Return hash for use as dict keys."""
    return hash(self.parts)

metaxy.FeatureKey.__eq__

__eq__(other: Any) -> bool

Check equality with another instance.

Source code in src/metaxy/models/types.py

def __eq__(self, other: Any) -> bool:
    """Check equality with another instance."""
    if isinstance(other, self.__class__):
        return self.parts == other.parts
    return super().__eq__(other)

metaxy.FeatureKey.to_column_suffix

to_column_suffix() -> str

Convert to a suffix usable for database column names (typically temporary).

Source code in src/metaxy/models/types.py

def to_column_suffix(self) -> str:
    """Convert to a suffix usable for database column names (typically temporary)."""
    return "__" + "_".join(self.parts)

metaxy.FieldKey

FieldKey(parts: str)

FieldKey(parts: Sequence[str])

FieldKey(parts: FieldKey)

FieldKey(parts: str | Sequence[str] | FieldKey)

Bases: _Key

Field key as a sequence of string parts.

Hashable for use as dict keys in registries. Parts cannot contain forward slashes (/) or double underscores (__).

Example:

```py
FieldKey("a/b/c")  # String format
# FieldKey(parts=['a', 'b', 'c'])

FieldKey(["a", "b", "c"])  # List format
# FieldKey(parts=['a', 'b', 'c'])

FieldKey(FieldKey(["a", "b", "c"]))  # FieldKey copy
# FieldKey(parts=['a', 'b', 'c'])
```

Source code in src/metaxy/models/types.py

def __init__(
    self,
    parts: str | Sequence[str] | FieldKey,
) -> None: ...

Attributes

metaxy.FieldKey.parts property

parts: tuple[str, ...]

Backward compatibility property for accessing root as parts.

metaxy.FieldKey.table_name property

table_name: str

Get SQL-like table name for this feature key.

Replaces hyphens with underscores for SQL compatibility.

Functions

metaxy.FieldKey.to_string

to_string() -> str

Convert to string representation with "/" separator.

Source code in src/metaxy/models/types.py

def to_string(self) -> str:
    """Convert to string representation with "/" separator."""
    return KEY_SEPARATOR.join(self.parts)

metaxy.FieldKey.to_struct_key

to_struct_key() -> str

Convert to a name that can be used as struct key in databases

Source code in src/metaxy/models/types.py

def to_struct_key(self) -> str:
    """Convert to a name that can be used as struct key in databases"""
    return "_".join(self.parts)

metaxy.FieldKey.to_column_suffix

to_column_suffix() -> str

Convert to a suffix usable for database column names (typically temporary).

Source code in src/metaxy/models/types.py

def to_column_suffix(self) -> str:
    """Convert to a suffix usable for database column names (typically temporary)."""
    return "__" + "_".join(self.parts)

metaxy.FieldKey.__repr__

__repr__() -> str

Return string representation.

Source code in src/metaxy/models/types.py

def __repr__(self) -> str:
    """Return string representation."""
    return self.to_string()

metaxy.FieldKey.__str__

__str__() -> str

Return string representation.

Source code in src/metaxy/models/types.py

def __str__(self) -> str:
    """Return string representation."""
    return self.to_string()

metaxy.FieldKey.__lt__

__lt__(other: Any) -> bool

Less than comparison for sorting.

Source code in src/metaxy/models/types.py

def __lt__(self, other: Any) -> bool:
    """Less than comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts < other.parts
    return NotImplemented

metaxy.FieldKey.__le__

__le__(other: Any) -> bool

Less than or equal comparison for sorting.

Source code in src/metaxy/models/types.py

def __le__(self, other: Any) -> bool:
    """Less than or equal comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts <= other.parts
    return NotImplemented

metaxy.FieldKey.__gt__

__gt__(other: Any) -> bool

Greater than comparison for sorting.

Source code in src/metaxy/models/types.py

def __gt__(self, other: Any) -> bool:
    """Greater than comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts > other.parts
    return NotImplemented

metaxy.FieldKey.__ge__

__ge__(other: Any) -> bool

Greater than or equal comparison for sorting.

Source code in src/metaxy/models/types.py

def __ge__(self, other: Any) -> bool:
    """Greater than or equal comparison for sorting."""
    if isinstance(other, self.__class__):
        return self.parts >= other.parts
    return NotImplemented

metaxy.FieldKey.__iter__

__iter__() -> Iterator[str]

Return iterator over parts.

Source code in src/metaxy/models/types.py

def __iter__(self) -> Iterator[str]:  # ty: ignore[invalid-method-override]
    """Return iterator over parts."""
    return iter(self.parts)

metaxy.FieldKey.__getitem__

__getitem__(index: int) -> str

Get part by index.

Source code in src/metaxy/models/types.py

def __getitem__(self, index: int) -> str:
    """Get part by index."""
    return self.parts[index]

metaxy.FieldKey.__len__

__len__() -> int

Get number of parts.

Source code in src/metaxy/models/types.py

def __len__(self) -> int:
    """Get number of parts."""
    return len(self.parts)

metaxy.FieldKey.__contains__

__contains__(item: str) -> bool

Check if part is in key.

Source code in src/metaxy/models/types.py

def __contains__(self, item: str) -> bool:
    """Check if part is in key."""
    return item in self.parts

metaxy.FieldKey.__reversed__

__reversed__()

Return reversed iterator over parts.

Source code in src/metaxy/models/types.py

def __reversed__(self):
    """Return reversed iterator over parts."""
    return reversed(self.parts)

metaxy.FieldKey.model_dump

model_dump(**kwargs: Any) -> Any

Serialize to string format for JSON dict key compatibility.

Source code in src/metaxy/models/types.py

def model_dump(self, **kwargs: Any) -> Any:
    """Serialize to string format for JSON dict key compatibility."""
    return self.to_string()

metaxy.FieldKey.__hash__

__hash__() -> int

Return hash for use as dict keys.

Source code in src/metaxy/models/types.py

def __hash__(self) -> int:
    """Return hash for use as dict keys."""
    return hash(self.parts)

metaxy.FieldKey.__eq__

__eq__(other: Any) -> bool

Check equality with another instance.

Source code in src/metaxy/models/types.py

def __eq__(self, other: Any) -> bool:
    """Check equality with another instance."""
    if isinstance(other, self.__class__):
        return self.parts == other.parts
    return super().__eq__(other)

Type Annotations

These are typically used to annotate function parameters. Most APIs in Metaxy accepts them and perform type coercion into canonical types.

metaxy.CoercibleToFeatureKey module-attribute

CoercibleToFeatureKey: TypeAlias = str | Sequence[str] | FeatureKey | type['BaseFeature']

metaxy.CoercibleToFieldKey module-attribute

CoercibleToFieldKey: TypeAlias = str | Sequence[str] | FieldKey

Pydantic Type Annotations

These types are used for type coercion into canonical types with Pydantic.

metaxy.ValidatedFeatureKey module-attribute

ValidatedFeatureKey: TypeAlias = FeatureKey

metaxy.ValidatedFieldKey module-attribute

ValidatedFieldKey: TypeAlias = FieldKey

metaxy.ValidatedFeatureKeySequence module-attribute

ValidatedFeatureKeySequence: TypeAlias = Sequence[ValidatedFeatureKey]

metaxy.ValidatedFieldKeySequence module-attribute

ValidatedFieldKeySequence: TypeAlias = Sequence[ValidatedFieldKey]

Adapters

These can perform type coercsion into canonical types in non-pydantic code.

metaxy.ValidatedFeatureKeyAdapter module-attribute

ValidatedFeatureKeyAdapter: TypeAdapter[ValidatedFeatureKey] = TypeAdapter(ValidatedFeatureKey)

metaxy.ValidatedFeatureKeySequenceAdapter module-attribute

ValidatedFeatureKeySequenceAdapter: TypeAdapter[ValidatedFeatureKeySequence] = TypeAdapter(ValidatedFeatureKeySequence)

metaxy.ValidatedFieldKeyAdapter module-attribute

ValidatedFieldKeyAdapter: TypeAdapter[ValidatedFieldKey] = TypeAdapter(ValidatedFieldKey)

metaxy.ValidatedFieldKeySequenceAdapter module-attribute

ValidatedFieldKeySequenceAdapter: TypeAdapter[ValidatedFieldKeySequence] = TypeAdapter(ValidatedFieldKeySequence)

Other Types

metaxy.models.types.SnapshotPushResult

Bases: NamedTuple

Result of recording a feature graph snapshot.

Attributes:

• snapshot_version (str) –

  The deterministic hash of the graph snapshot

• already_pushed (bool) –

  True if this snapshot_version was already pushed previously

• updated_features (list[str]) –

  List of feature keys with updated information (changed full_definition_version)

metaxy.IDColumns module-attribute

IDColumns: TypeAlias = Sequence[str]