Metaxy Integrations

Orchestration

• Dagster

  ---

  Orchestration • Data Platform

  Seamlessly integrate Metaxy with Dagster with the power of @metaxify and the MetaxyIOManager.

  Recommended

  Metaxy has been built with Dagster in mind. This integration is the best way to organize, materialize and observe multiple Metaxy features at scale.

  Integration docs

  API docs

Metadata Stores

Learn more about metadata stores here.

• BigQuery

  ---

  Database

  Use Google BigQuery - scalable serverless analytical database on GCP.

  Integration docs

  API docs

• ClickHouse

  ---

  Database

  Leverage the lightning-fast analytical ClickHouse database for large metadata volume and high-throughput setups.

  Recommended

  Ideal for production.

  Integration docs

  API docs

• Delta Lake

  ---

  Storage

  Store metadata in Delta Lake format in local files or remote object stores (S3, GCS, and others). (1)

  Recommended

  Ideal for dev environments.

  Integration docs

  API docs

• DuckDB

  ---

  Database • Storage

  Use DuckDB - a fast analytical database with support for local and remote compute. DuckLake is available as well.

  Warning

  DuckDB is not recommended for production due to parallel writes limitations.

  Integration docs

  API docs

• LanceDB

  ---

  Database • Storage

  Use the multi-modal LanceDB database or Lance storage format. (2)

  Integration docs

  API docs

1. uses a local versioning engine implemented in Polars and polars-hash

2. uses a local versioning engine implemented in Polars and polars-hash

Plugins

• SQLAlchemy

  ---

  ORM • Database

  Retrieve SQLAlchemy URLs and MetaData for the current Metaxy project from Metaxy MetadataStore objects.

  Integration docs

  API docs

• SQLModel

  ---

  ORM • Database

  Adds SQLModel capabilities to metaxy.BaseFeature class.

  Integration docs

  API docs