Reference data¶
Yeliztli annotates your variants against public scientific datasets. They fall into two groups: prebuilt bundles that Yeliztli publishes, and pipeline sources fetched from their original providers. They also have two setup roles:
- Required databases must be downloaded or built and pass integrity checks before the first-time setup wizard can complete.
- Optional sources add context or optional features. Most can be installed later from Settings → Database Management; manual or bring-your-own sources use the local ingest path documented for that source.
The setup-blocking required databases are ClinVar, gnomAD, dbNSFP, CPIC,
GWAS Catalog, dbSNP, and MONDO/HPO. If any required database is Failed or not
integrity-ready, setup remains active and the dashboard is not reachable until you resume,
retry, or clean that database. Reference data is stored under your data directory (default
~/.yeliztli/).
All downloads are resumable and integrity-checked. You can inspect, resume, verify, or clean any of them under Settings → System Health → Database Health.
First-run duration¶
Full reference-data setup is a long one-time operation. It uses more than 60 GB at peak, with ~80 GB recommended for headroom, and commonly takes on the order of an hour or more. Slow networks, slower disks, or optional databases can make it considerably longer.
The dbNSFP step dominates the runtime. Its source archive is a large network download,
and then Yeliztli parses, builds, and indexes a multi-GB SQLite database. Download progress is
shown as bytes transferred; the later build/index phase is CPU- and disk-bound and may not
show the same moving progress bar. If Settings → System Health → Database Health shows
the database as Downloading or Building, setup is still active. If it shows Failed, use
the repair/resume controls from that page.
Prebuilt bundles¶
These are published as GitHub release assets, pinned by version and SHA-256 checksum in
bundles/manifest.json.
| Bundle | Setup role | Approx. size | What it provides |
|---|---|---|---|
| gnomAD allele frequencies | Required | ~1.30 GB download / ~2.85 GB installed | Population allele frequencies, observed allele counts, and homozygous counts — CC0 / public domain. |
| VEP consequence bundle | Optional | ~360 MB | Pre-computed variant consequences, HGVS, and transcript context for the genotyped sites. |
| PGS scores | Optional | ~104 MB | Polygenic-score weight sets used by the risk modules. |
| Ancestry PCA bundle | Optional (ships with app) | ~0.4 MB | Ancestry-informative markers and PCA loadings — ships inside the app, no download. |
| Ancestry LAI bundle | Optional | ~1.7 GB | Local-ancestry-inference models + phasing reference for Tier-2 chromosome painting. Requires Java 8+; only download it if you want chromosome-level ancestry. |
Pipeline sources¶
These are downloaded from the original providers. Each retains its own license — the full
attribution list lives in the repository
NOTICE file.
| Source | Setup role | Purpose | Approx. setup footprint | License |
|---|---|---|---|---|
| ClinVar (NCBI) | Required | Clinical variant classifications | ~250 MB | Public domain |
| dbNSFP | Required | In-silico pathogenicity predictions (REVEL, CADD, ...) | ~50 GB transient ZIP + ~10+ GB built DB | Academic / non-commercial |
| CPIC | Required | Pharmacogenomics allele & guideline data | ~5 MB | CC0-1.0 |
| ClinGen | Optional | Gene-disease validity & dosage | ~1 MB | CC0-1.0 |
| PharmVar | Supporting source for CPIC | Pharmacogene star-allele definitions | Small metadata source | Open |
| AlphaMissense | Optional | Missense pathogenicity predictions | ~3.5 GB when installed | CC-BY-4.0 |
| GWAS Catalog (EBI) | Required | Trait/disease associations for risk modules | ~100 MB | Open |
| dbSNP (NCBI) | Required | rsID merge/identity resolution | ~20 MB | Public domain |
| MONDO/HPO (Monarch) | Required | Disease-scoped phenotype associations | ~34 MB per source bundle | Source-specific terms |
| PharmGKB | Optional context | Clinical drug annotations | Small metadata source | CC-BY-SA-4.0 |
| FDA drug labels (via PharmGKB) | Optional context | Pharmacogenomic labeling | Small metadata source | CC-BY-SA-4.0 |
| GTEx eQTL | Optional | Tissue eQTLs for functional context | ~3 GB when installed | Open-access summary stats |
| SpliceAI | Optional / manual | User-supplied splice-effect prediction database | Depends on local ingest | Illumina non-commercial terms |
| ENCODE cCREs | Optional | Candidate cis-regulatory elements for Genome Browser tracks | ~30 MB when installed | ENCODE data terms |
UCSC hg19 FASTA + RefSeq (refGene) |
Optional local browser reference | Fully local Genome Browser reference and gene track | ~4 GB when installed | UCSC Genome Browser data terms |
For MONDO/HPO, phenotype terms and inheritance are attached to a MONDO disease only through
an authoritative skos:exactMatch cross-reference. Unmatched or ambiguous source disease
identifiers remain unannotated rather than being copied to another disease for the same gene.
Each successful refresh transfers about 34 MB across the Monarch primary archive, HPO export,
and MONDO SSSOM mapping. The downloader retains every successfully validated source bundle as
append-only provenance rather than recursively deleting a directory after a path identity check.
Incomplete, malformed, or operator-created directories are also never automatically removed;
inspect all bundle directories manually before any operator-directed cleanup.
A MONDO/HPO installation carried over from a release that predates the disease-scoped ingestion is withheld from every phenotype lookup, because its rows cannot be proven to be scoped to the disease they were recorded against. Database health reports such an install as Partial rather than Ready and setup keeps you on the Databases step until it is rebuilt, so an upgraded machine is never admitted while phenotype annotations would come back empty. Refresh MONDO/HPO to restore disease-scoped rows.
The loader rejects grossly truncated inputs before replacing installed rows: the compressed
Monarch primary archive must be at least 100,000 bytes, the HPO export at least 10,000,000
bytes, and the MONDO SSSOM export must yield at least 50,000 unambiguous exact mappings. These
are fail-closed operational guards based on the recorded source snapshot, not proof of semantic
completeness. Configure the downloads directory and its mondo_hpo_sources child as private to
the updating account and not group/world-writable, and prefer ancestors that are likewise not
writable by group or other users.
These directory-trust conditions are advisory today, not enforced. The loader inspects the
downloads root and every lexical and resolved ancestor for ownership and group/world
writability, and records what it finds — mondo_hpo_source_bundle_directory_not_private and
mondo_hpo_downloads_ancestor_not_trusted, the latter carrying a reason of owner or
group_or_world_writable — but it then proceeds with the build. Treat a clean log as evidence
the namespace was private, and treat those warnings as the signal it was not; do not read the
absence of a failure as proof of enforcement. Restoring enforcement is tracked in issue #2316.
They were made advisory because MONDO/HPO is a required database, so refusing a directory did
not degrade the install, it prevented setup from completing at all — including on ordinary
configurations such as a cooperative umask of 0002, a provisioned group-writable mount, or a
restored directory owned by another account.
What still fails closed is everything about the loader's own files rather than the operator's filesystem: descriptor pinning against path substitution, staging identity checks, the source content hashes below, and the manifest verification before a bundle is published. The loader hashes its held source files before parsing and verifies those same bytes again before it writes the provenance manifest or publishes a bundle. Descriptor checks and the final pre-commit validation detect swaps that the process can observe, but cannot make a filesystem rename atomic with the SQLite commit; an administrator, the platform temporary-directory service, or another same-privilege principal remain outside that boundary.
dbNSFP license
dbNSFP is distributed under an academic / non-commercial license. Make sure your use complies with its terms. Its setup footprint is also large: the source archive is removed after a successful build, but an interrupted build may keep the completed archive so setup can resume without starting another large download. The build and index step can take a long time after the archive has already downloaded.
HIBAG HLA model files¶
The HLA (imputed) feature uses HIBAG pre-fit model files, but those files are not Yeliztli release bundles or setup-wizard downloads. They are bring-your-own external inputs because HIBAG itself is an operator-installed R/Bioconductor runtime and the model files carry their own distribution terms.
If you enable HLA imputation, fetch the model files yourself, place them in the directory
configured by YELIZTLI_HIBAG_MODEL_DIR, and keep their expected names:
European-HLA4.RData, Asian-HLA4.RData, Hispanic-HLA4.RData, or African-HLA4.RData.
Then run scripts/predict_hla.py for each sample to populate hla_calls.
Genome Browser local reference files¶
The Genome Browser can run without contacting IGV.js reference hosts when these local files are installed:
grch37.fagrch37.fa.faigrch37_refseq.bedgenome_browser_reference_manifest.json
By default Yeliztli looks for them in the data directory. You can point to local runtime files
with YELIZTLI_GRCH37_FASTA_PATH and
YELIZTLI_GENOME_BROWSER_REFSEQ_TRACK_PATH, but the files are accepted only when the manifest
describes the expected UCSC hg19 FASTA / refGene build and the FASTA index matches GRCh37/hg19
sentinel chromosome lengths. If any file is missing or validation fails, the Genome Browser keeps
the disclosure-gated hosted hg19 fallback.
Maintainers can build the local reference files from UCSC sources with
scripts/build_genome_browser_reference.py.
The SLURM procedure and provenance checklist are in the
Genome Browser reference bundle runbook.
Updating reference data¶
The per-database auto-update table in Settings → Database Management covers the sources registered in the update manager: ClinVar, dbNSFP, CPIC, GWAS Catalog, dbSNP, MONDO/HPO, ENCODE cCREs, the published gnomAD, VEP, LAI, and PGS score bundles, and the app-shipped ancestry PCA bundle. For those sources you can configure auto-update toggles and an optional bandwidth window for large downloads. See updating.
Static / manual-refresh sources¶
Some registered sources are install-time or bring-your-own context layers, not auto-updated reference data:
- AlphaMissense is built when installed and is not version-checked by the automatic update system.
- GTEx eQTL is built when installed and is not version-checked by the automatic update system.
- ClinGen gene-disease validity data is built when installed and is not version-checked by the automatic update system.
- SpliceAI is a manual, user-supplied database and is never downloaded or updated by Yeliztli.
These sources do not appear in the per-database auto-update table and do not have automatic upstream release checks. To refresh them, use the relevant manual rebuild, local ingest, or setup flow; then re-annotate samples whose results should reflect the refreshed snapshot.