Skip to content

Metadata Stores

Metadata stores may come in two flavors.

Database-Backed

These metadata stores provide external compute resources. The most common example of such stores is databases. Metaxy delegates all versioning computations and operations to external compute as much as possible. (1)

  1. 🔥 Typically (1) the entire MetadataStore.resolve_update can be executed externally!

  2. Except the cases enumerated in [../../guide/learn/metadata-stores.md]

These metadata stores can be found here.

Warning

Metaxy does not handle infrastructure setup. Make sure to have large tables partitioned as appropriate for your use case.

Example

ClickHouse is an excellent choice for a production metadata store.

Tip

Some of them such as LanceDB or DuckDB can also act as local compute engines.

Storage Only

These metadata stores only provide storage and rely on local (also referred to as embedded) compute.

The available storage-only stores can be found here.

Example

DeltaLake is an excellent choice for a storage-only metadata store.

Choosing the Right Metadata Store

Compute-backed stores are typically more performant, but require additional infrastructure and maintenance.

For production environments that need to handle big metadata volumes, consider database-backed stores.

For development, testing, branch deployments, and other scenarios where you want to keep things simple, consider using a storage-only store.

Warning

Not all metadata stores support parallel writes. For example, DuckDB requires application level work-arounds.

Reference