sgkit.io.vcf.vcf_to_zarr

sgkit.io.vcf.vcf_to_zarr(input, output, *, target_part_size='auto', regions=None, chunk_length=10000, chunk_width=1000, compressor=Blosc(cname='zstd', clevel=7, shuffle=AUTOSHUFFLE, blocksize=0), encoding=None, temp_chunk_length=None, tempdir=None, tempdir_storage_options=None, ploidy=2, mixed_ploidy=False, truncate_calls=False, max_alt_alleles=3, fields=None, exclude_fields=None, field_defs=None, read_chunk_length=None, retain_temp_files=None)

Convert VCF files to a single Zarr on-disk store.

By default, the conversion is carried out in parallel, by writing the output for each part to a separate, intermediate Zarr store in tempdir. Then, in a second step the intermediate outputs are concatenated and rechunked into the final output Zarr store in output.

Conversion is carried out sequentially if target_part_size is None, and regions is None.

For more control over these two steps, consider using vcf_to_zarrs() followed by concat_zarrs().

Parameters

input : str | Path | Sequence[Union[str, Path]]Union[str, Path, Sequence[Union[str, Path]]]

A path (or paths) to the input BCF or VCF file (or files). VCF files should be compressed and have a .tbi or .csi index file. BCF files should have a .csi index file.

output : str | Path | MutableMapping[str, bytes]Union[str, Path, MutableMapping[str, bytes]]

Zarr store or path to directory in file system.

target_part_size : None | int | strUnion[None, int, str] (default: 'auto')

The desired size, in bytes, of each (compressed) part of the input to be processed in parallel. Defaults to "auto", which will pick a good size (currently 20MB). A value of None means that the input will be processed sequentially. The setting will be ignored if regions is also specified.

regions : None | Sequence[str] | Sequence[Optional[Sequence[str]]]Union[None, Sequence[str], Sequence[Optional[Sequence[str]]]] (default: None)

Genomic region or regions to extract variants for. For multiple inputs, multiple input regions are specified as a sequence of values which may be None, or a sequence of region strings. Takes priority over target_part_size if both are not None.

chunk_length : int (default: 10000)

Length (number of variants) of chunks in which data are stored, by default 10,000.

chunk_width : int (default: 1000)

Width (number of samples) to use when storing chunks in output, by default 1,000.

compressor : Any | NoneOptional[Any] (default: Blosc(cname='zstd', clevel=7, shuffle=AUTOSHUFFLE, blocksize=0))

Zarr compressor, by default Blosc + zstd with compression level 7 and auto-shuffle. No compression is used when set as None.

encoding : Any | NoneOptional[Any] (default: None)

Variable-specific encodings for xarray, specified as a nested dictionary with variable names as keys and dictionaries of variable specific encodings as values. Can be used to override Zarr compressor and filters on a per-variable basis, e.g., {"call_genotype": {"compressor": Blosc("zstd", 9)}}.

temp_chunk_length : int | NoneOptional[int] (default: None)

Length (number of variants) of chunks for temporary intermediate files. Set this to be smaller than chunk_length to avoid memory errors when loading files with very large numbers of samples. Must be evenly divisible into chunk_length. Defaults to chunk_length if not set.

tempdir : str | Path | NoneUnion[str, Path, None] (default: None)

Temporary directory where intermediate files are stored. The default None means use the system default temporary directory.

tempdir_storage_options : {str: str} | NoneOptional[Dict[str, str]] (default: None)

Any additional parameters for the storage backend for tempdir (see fsspec.open).

ploidy : int (default: 2)

The (maximum) ploidy of genotypes in the VCF file.

mixed_ploidy : bool (default: False)

If True, genotype calls with fewer alleles than the specified ploidy will be padded with the fill (non-allele) sentinel value of -2. If false, calls with fewer alleles than the specified ploidy will be treated as incomplete and will be padded with the missing-allele sentinel value of -1.

truncate_calls : bool (default: False)

If True, genotype calls with more alleles than the specified (maximum) ploidy value will be truncated to size ploidy. If false, calls with more alleles than the specified ploidy will raise an exception.

max_alt_alleles : int (default: 3)

The (maximum) number of alternate alleles in the VCF file. Any records with more than this number of alternate alleles will have the extra alleles dropped (the variant_allele variable will be truncated). Any call genotype fields with the extra alleles will be changed to the missing-allele sentinel value of -1.

fields : Sequence[str] | NoneOptional[Sequence[str]] (default: None)

Extra fields to extract data for. A list of strings, with INFO or FORMAT prefixes. Wildcards are permitted too, for example: ["INFO/*", "FORMAT/DP"].

field_defs : {str: {str: Any}} | NoneOptional[Dict[str, Dict[str, Any]]] (default: None)

Per-field information that overrides the field definitions in the VCF header, or provides extra information needed in the dataset representation. Definitions are a represented as a dictionary whose keys are the field names, and values are dictionaries with any of the following keys: Number, Type, Description, dimension. The first three correspond to VCF header values, and dimension is the name of the final dimension in the array for the case where Number is a fixed integer larger than 1. For example, {"INFO/AC": {"Number": "A"}, "FORMAT/HQ": {"dimension": "haplotypes"}} overrides the INFO/AC field to be Number A (useful if the VCF defines it as having variable length with .), and names the final dimension of the HQ array (which is defined as Number 2 in the VCF header) as haplotypes. (Note that Number A is the number of alternate alleles, see section 1.4.2 of the VCF spec https://samtools.github.io/hts-specs/VCFv4.3.pdf.)

read_chunk_length : int | NoneOptional[int] (default: None)

Length (number of variants) of chunks to read from the VCF file at a time. Use this option to reduce memory usage by using a value lower than chunk_length with a small cost in extra run time. The increase in runtime becomes higher as the ratio of read_chunk_length to Defaults to None, which means that a value equal to chunk_length is used. The memory usage of the conversion process is proportional to read_chunk_length*n_samples*(1+n_ploidy) so this option is mainly useful for very large numbers of samples and/or where a large chunk_size is desirable to reduce the number of dask tasks needed in downstream analysis.

retain_temp_files : bool | NoneOptional[bool] (default: None)

If True, intermediate files are retained after the final output is written. Defaults to deleting intermediate files. Intermediate files are deleted in a single process, so for large VCF files this can be slow.

Return type

None