Storage backends

Storage backends#

Vcztools opens VCZ datasets through one of four Zarr storage backends, selected via the --backend-storage CLI option or the backend_storage keyword argument to vcztools.open_zarr(). The matching --storage-option KEY=VALUE (CLI, repeatable) or storage_options dict (Python) forwards configuration to the underlying library; the keys it accepts depend on the backend (and, for fsspec, on the URL scheme).

Backend

Selector

Best for

local (default)

omit --backend-storage

Local files and .zip archives

fsspec

--backend-storage fsspec

Broadest cloud coverage via s3fs, gcsfs, adlfs, …

obstore

--backend-storage obstore

High-throughput object-store reads (Rust object_store under the hood)

icechunk

--backend-storage icechunk

Versioned Icechunk repositories on local disk, S3, or Azure

The obstore and icechunk backends are install extras: pull them in with pip install vcztools[obstore] or pip install vcztools[icechunk] when you need them. Requesting one without the corresponding package installed raises ImportError with the matching pip command. The fsspec backend and its filesystem packages (s3fs, gcsfs, adlfs, …) stay separate installs because the protocol you need depends on where your data lives.

Local (default)#

With no --backend-storage option, vcztools opens local data only:

  • .zip paths are opened as a zarr.storage.ZipStore.

  • Directory paths are opened as a zarr.storage.LocalStore.

  • Remote URLs (any string containing ://) and non-empty storage_options raise — pick an explicit backend for those.

vcztools view sample.vcz
vcztools view sample.vcz.zip

Definitive docs: Zarr storage.

fsspec#

vcztools view --backend-storage fsspec s3://<bucket>/path/to.vcz

Routes through zarr.storage.FsspecStore via FsspecStore.from_url. Local paths and pathlib.Path inputs are auto-promoted to file:// URIs.

storage_options is forwarded to the fsspec filesystem constructor that fsspec selects from the URL scheme. The accepted keys therefore depend on the protocol — see the per-protocol documentation for the exhaustive list:

  • S3: s3fs (e.g. key, secret, endpoint_url, anon).

  • GCS: gcsfs (e.g. token, project).

  • Azure: adlfs (e.g. account_name, account_key, sas_token).

  • HTTP(S): fsspec http (e.g. client_kwargs).

Fsspec is included in the default install of vcztools, but additional protocols must be installed separately. For example, to use S3 you need to install s3fs:

python3 -m pip install s3fs

Definitive docs: fsspec.

obstore#

vcztools view --backend-storage obstore s3://<bucket>/path/to.vcz

Routes through zarr.storage.ObjectStore built from obstore.store.from_url.

storage_options is unpacked as keyword arguments to obstore.store.from_url. Common keys:

  • client_options — request-level options (timeouts, headers, TLS).

  • retry_config — backoff and jitter for retries.

  • Scheme-specific credentials, e.g. aws_access_key_id, aws_secret_access_key, aws_region for S3; azure_storage_account_name, azure_storage_account_key for Azure.

Install:

python3 -m pip install vcztools[obstore]

Definitive docs: obstore and the store.from_url API reference.

icechunk#

vcztools view --backend-storage icechunk s3://<bucket>/repo

Opens an Icechunk repository’s main branch as a read-only Zarr session. The URL scheme picks the storage constructor:

  • Local paths → icechunk.Storage.new_local_filesystem.

  • s3://… → icechunk.s3_storage(..., from_env=True).

  • az://…, azure://…, abfs://…, abfss://…, and Azure https://…blob.core.windows.net URLs → icechunk.azure_storage(..., from_env=True).

storage_options is forwarded as keyword arguments to the chosen constructor (e.g. region, endpoint_url for s3_storage). Local paths reject non-empty storage_options.

Install:

python3 -m pip install vcztools[icechunk]

Definitive docs: Icechunk.

Example: read from S3 with fsspec#

Set up credentials (e.g. via environment variables described in the s3fs documentation):

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

Then run vcztools against an s3:// URL:

vcztools view --backend-storage fsspec s3://<bucket>/path/to.vcz

Equivalent in Python:

import vcztools

root = vcztools.open_zarr(
    "s3://<bucket>/path/to.vcz", backend_storage="fsspec"
)