TileDB-VCF

When to use

Use for cohort variant storage, incremental sample ingestion, interval queries, and exporting subsets for downstream genomics. TileDB-VCF stores records; it does not perform joint variant calling, association testing, normalization, or population structure adjustment. Dataset size alone does not determine whether cloud execution is appropriate: benchmark the intended sample, region, and attribute workload.

This skill targets released TileDB-VCF 0.40.3. Native Python/CLI checks used two synthetic single-sample VCFs on macOS ARM64. Cloud client 0.14.4 contracts were checked against its released wheel and mocked dispatch; hosted queries and jobs were not executed. See installation and verification notes.

Install and verify the native stack

The tiledbvcf Python distribution is not published on PyPI at this review. The official tiledb Conda channel supplies tiledbvcf-py 0.40.3 for macOS ARM64, macOS x86-64, and Linux x86-64, with Python 3.9-3.12 builds. tiledb on PyPI is TileDB-Py and does not install TileDB-VCF. Native Apple Silicon no longer requires forcing CONDA_SUBDIR=osx-64.

For the tested macOS ARM64 environment, pins below avoid versioned-library import failures in the otherwise successful current Conda solve:

conda create -n tiledb-vcf -c tiledb -c conda-forge \
  python=3.12 tiledbvcf-py=0.40.3 \
  azure-core-cpp=1.16.2 azure-storage-blobs-cpp=12.16.0 \
  azure-storage-files-datalake-cpp=12.14.0 capnproto=1.4.0 c-blosc2=2.23.1
conda activate tiledb-vcf
python -c 'import tiledbvcf; print(tiledbvcf.version)'
tiledbvcf version

The equivalent Micromamba solve/install was executed. These extra ABI pins are a verified macOS ARM64 workaround, not a claim about all platforms. Preserve the resolved environment for production. Other-platform installation and official Docker images are alternatives documented upstream, not tested here.

Prepare and create a cohort

1. Record reference assembly, contig naming/lengths, callers, normalization rules, sample identity, and file checksums. Do not mix assemblies or assume 1 and chr1 are equivalent. Confirm compatible headers across inputs.
2. Each input must contain one sample, be coordinate sorted, and have an index. Use BGZF-compressed VCF or BCF with a matching .csi/.tbi. Inspect bcftools query -l sample1.vcf.gz; filenames are not sample identifiers.
3. Create the dataset explicitly, then ingest. Opening mode="w" alone does not create its schema. Materialize frequently read fields at creation time.
4. Validate a small known interval and a round-trip export before scaling.

For existing sorted plain-text VCFs (run once per input; output names must be new):

bcftools view -Oz -o sample1.vcf.gz sample1.vcf
bcftools index -c sample1.vcf.gz
bcftools view -Oz -o sample2.vcf.gz sample2.vcf
bcftools index -c sample2.vcf.gz

The following local examples were exercised with sample IDs S1 and S2, contig chr1, and coordinates 1-100; substitute the verified IDs and intervals in real data.

import tiledbvcf

uri = "cohort"
config = {"sm.compute_concurrency_level": "2", "sm.io_concurrency_level": "2"}
with tiledbvcf.Dataset(uri, mode="w", tiledb_config=config) as ds:
    ds.create_dataset(extra_attrs=["fmt_GT", "fmt_DP"])
    ds.ingest_samples(
        ["sample1.vcf.gz"], threads=2, total_memory_budget_mb=512
    )

# Incremental ingestion uses the existing dataset; do not call create_dataset again.
with tiledbvcf.Dataset(uri, mode="w", tiledb_config=config) as ds:
    ds.ingest_samples(
        ["sample2.vcf.gz"], threads=2, total_memory_budget_mb=512
    )

The schema-v4 data array has contig, start-coordinate, and sample dimensions; headers and optional statistics are separate arrays in the dataset group. Fields not explicitly materialized remain in the INFO/FORMAT payloads. Query names such as pos_start need not match raw storage column names.

Upstream supports parallel thread/process ingestion. Assign distinct sample work and coordinate lifecycle/maintenance operations; do not blindly re-ingest samples. resume=True supports interrupted ingestion, not arbitrary duplicate correction.

Query completely and interpret coordinates correctly

…