Metaxy + Delta Lake

Metaxy implements DeltaMetadataStore that stores metadata in Delta Lake (also known as a LakeHouse format) and uses an in-memory Polars versioning engine.

It supports the local filesystem and remote object stores.

Installation

pip install 'metaxy[delta]'

Using Object Stores

Point root_path at any supported URI (s3://, abfss://, gs://, ...) and forward credentials with storage_options. The dict is passed verbatim to deltalake.

Learn more in the API docs.

Storage Layout

It's possible to control how feature keys map to DeltaLake table locations with the layout parameter:

• nested (default) places every feature in its own directory: your/feature/key.delta
• flat stores all of them in the same directory: your__feature_key.delta

Reference

• Configuration
• API