gtdbprep: classify genomes without the 100 GB download
gtdbprep is the laptop half of a split GTDB-Tk. It runs GTDB-Tk’s own
light stages (identify, then align --skip_gtdb_refs) on your genomes, after giving them and
their contigs opaque names. It packs the results into a bundle of a few kilobytes per genome. The
heavy step, tree placement, can then run elsewhere: on a machine with the full reference package,
on a laptop with gPlace and a memory cap, or on a shared server if one is ever built.
gtdbtk classify --align_dir <bundle> reads the bundle unchanged.
It wraps GTDB-Tk rather than reimplementing it. Every number in the bundle comes from GTDB-Tk 2.8.0 itself, run as a subprocess.
GTDB-Tk’s reference data for R232 is a 61 GB download that unpacks to about 100 GB, and bacterial placement needs about 140 GB of RAM (see GTDB-Tk and memory). Gene calling, marker search and alignment need none of that. They need 77 MB of marker HMMs, masks and metadata, and take well under a minute per genome on a laptop. Splitting there means:
- the laptop never downloads the 100 GB package;
- only a masked protein alignment and marker counts leave the laptop, never the DNA;
- the result is still GTDB-Tk’s own result, as the round trip below checks.
# Laptop: GTDB-Tk 2.8 in a conda/micromamba env, plus the 77 MB client reference datagtdbprep prep --genomes mags/ -x fa --work prep-work --out bundle.tar.gz \ --data r232-client/ --env ~/micromamba/envs/gtdbtkirit-gtdb-prep ... # the same as `gtdbprep prep ...`
# Optional: also add skani 0.3.1 sketches for a remote ANI screen (contain DNA, see Privacy)gtdbprep prep ... --sketch --skani /path/to/skani-0.3.1
# Where the full reference data is: classify the bundle, results get the original names backgtdbprep classify --bundle bundle.tar.gz --work prep-work --data release232/
# Testing: prep, classify from the bundle, plain classify_wf, compare genome by genomegtdbprep verify --genomes mags/ -x fa --work rt-work --data release232/ --env ...
# Laptop: species by ANI, then tree placement with gPlace for the rest (sections below)gtdbprep ani --genomes mags/ -x fa --work ani-work ...gtdbprep place --genomes mags/ -x fa --work place-work --skip-assigned ani-work \ --data placement-lite/release232 --env ~/micromamba/envs/gtdbtkOptions shared by all commands: --data (or GTDBTK_DATA_PATH), --env (or GTDBPREP_ENV, a
conda prefix whose bin/ goes first on PATH), --cpus (default 4). --full-tree true|false on
classify, verify and place defaults to false, or to true when the reference data has an empty
split/ folder (GTDB’s mockup database).
prep writes into --work:
| Path | What | Leaves the laptop? |
|---|---|---|
renamed/<id>.fna |
genomes with opaque names (sequence bytes unchanged) | no |
mapping.tsv |
opaque ID → GTDB-Tk genome ID, source path, source SHA-256, contig count, and why an input was left out (if it was) | no, never |
contigs/<id>.tsv |
opaque contig name → original header | no |
gtdbtk/ |
GTDB-Tk’s full identify/align output (genes, HMM hits) |
no |
logs/ |
one log per command | no |
bundle/ |
the bundle (also copied to --out) |
yes, this is what you send |
On Unix the work folder is made owner-only (mode 0700), and mapping.tsv and contigs/*.tsv are
written with mode 0600.
Input files that are empty or not FASTA (no > header, or sequence before the first one) are left
out with a warning instead of stopping the run. mapping.tsv lists them with opaque ID - and the
reason; prep prints them; the manifest records only how many there were (inputs_excluded), not
their names. classify repeats the warning, and verify reports them as excluded (not as
differences) and runs classify_wf on the remaining inputs only.
Spaces in paths. GTDB-Tk 2.8.0 refuses genome paths that contain a space (“unsupported by
downstream applications”), and it reads the renamed genomes from the work folder, so prep and
classify refuse a work folder with a space in its path before doing anything. The input folder
may have spaces: only gtdbprep reads it, and verify gives classify_wf a folder of links to the
inputs inside the work folder. A --data path with a space is untested.
--out takes a folder, a .tar, or a .tar.gz. It is checked before anything runs: it must not
exist yet (an empty folder is fine) and its parent folder must exist.
What is reproducible. Tar files have fixed metadata (mtime 0, mode 644, owner 0, sorted
entries), and every gzip file in the bundle is written with a fixed header (no file name, mtime 0;
GTDB-Tk’s own .gz files are decompressed and recompressed so its header, which names the source
file and the time it ran, does not leave the laptop). So the same bundle folder always gives the
same .tar bytes. Two prep runs on the same genomes do not give the same bundle: the opaque IDs
are random and the manifest records started and created times. .tar.gz bytes also depend on
the gzip compressor (flate2 and its backend version), so a different gtdbprep build can compress
the same .tar to different bytes; compare .tar files or the manifest’s files hashes instead.
Bundle format (gtdbprep-bundle/1)
Section titled “Bundle format (gtdbprep-bundle/1)”The bundle uses GTDB-Tk’s own folder layout and file names (prefix gtdbtk), so it is a valid
--align_dir:
manifest.jsonidentify/gtdbtk.bac120.markers_summary.tsv marker counts and lists per genomeidentify/gtdbtk.ar53.markers_summary.tsvidentify/gtdbtk.translation_table_summary.tsv genetic code per genome (11 or 4)identify/gtdbtk.failed_genomes.tsv Prodigal failures (often empty)align/gtdbtk.bac120.user_msa.fasta.gz masked alignment, 5,010 columnsalign/gtdbtk.ar53.user_msa.fasta.gz masked alignment, 4,369 columns (if any archaea)align/gtdbtk.{bac120,ar53}.filtered.tsv genomes below 10% of columnsalign/intermediate_results/gtdbtk.{align,identify}.failed.tsv (when GTDB-Tk writes them)sketches/<id>.fna.sketch only with --sketchmanifest.json records:
format,client(name, version),startedandcreated(RFC 3339, UTC);gtdb_release(frommetadata/metadata.txt) andtools: the version line each tool printed (gtdbtk,prodigal,hmmer, andskaniwith--sketch). A line containing/or\(a Python traceback naming a path, say) is not kept:reportedis then just the parsed version;commands: the exact GTDB-Tk and skani argument lists, with local paths replaced by<work>and<GTDBTK_DATA_PATH>;reference: formarkers/,masks/,metadata/andtaxonomy/, GTDB-Tk’s own folder hash (sha1_dir) next to theREF_HASHESvalue GTDB-Tk 2.8.0 expects for that release, plus a SHA-256 over the folder’ssha256sumlisting, and per-file SHA-256 for every folder exceptmarkers/(182 files, covered by the listing hash).preprefuses data whose markers, masks or metadata don’t matchREF_HASHES(--allow-ref-mismatchoverrides). See Reference data fortaxonomy/;genomes: per opaque ID, the SHA-256 of the renamed FASTA, contig count, domain call, single-copy marker counts, genetic code, which MSA holds it, and why GTDB-Tk excluded it if it did;inputs_excluded: how many input filesprepleft out (empty or not FASTA), without names;sketches: skani version and parameters (c = 125, marker c = 1000, k = 15) and the warning text;files: size and SHA-256 of every other file in the bundle.classifyrejects a bundle whose files don’t match, and checks again after GTDB-Tk has run that nothing was changed;privacy: the statements below.
Genomes are listed in the order of their random IDs, so the bundle doesn’t reveal the original file order.
Privacy
Section titled “Privacy”Leaves the laptop (the bundle): for each genome, a masked alignment of 120 bacterial (or 53
archaeal) marker proteins (about 5,000 amino acids); marker counts and the list of which markers
were found; the genetic code; the domain call; the contig count; the SHA-256 of the renamed FASTA.
Also the tool versions, reference-data hashes, and the number of inputs left out. The contig count
is a mild identifying feature: with the genome size class it can help match a bundle to a known
assembly. The bundle’s own privacy text says so.
Stays on the laptop: DNA sequences (except the k-mer samples in sketches, if you add them), genes and proteins outside the markers, file names, contig names, folder paths, and the mapping from opaque IDs back to your genomes.
Leak scan. After the manifest is written, prep searches every bundle file, manifest.json
included, decompressed, for: the original genome IDs, file names and contig headers (whole header
and first word); the work, data and input folders and every input file path, each as given, made
absolute, and with symlinks resolved; $HOME in the same three forms; and, as whole words only,
$USER, $LOGNAME and the host name. With --out, it scans what it wrote again (for a .tar or
.tar.gz, the tar headers and every entry, decompressed). On any hit it deletes the bundle (or the
--out copy) and fails, so a bundle that fails the scan cannot be sent by mistake.
Names and paths shorter than 6 characters (user and host names: shorter than 3) are not scanned
for, because they would match by chance inside alignments and marker lists. They cannot reach the
bundle by construction: GTDB-Tk only ever sees the renamed files (<random id>.fna, contigs
<id>_c<n>), so none of its outputs can contain an original name; the bundle files are GTDB-Tk’s
outputs plus a manifest that gtdbprep builds from opaque IDs, counts and hashes; and the commands
recorded in it have local paths replaced by <work> and <GTDBTK_DATA_PATH>. The scan is a
second check of that, not the mechanism.
What the alignment still says. It identifies the organism; that is its purpose. For a novel lineage it contains partial sequences of conserved housekeeping proteins, which may matter before publication. The FASTA hash lets someone who already has the same genome confirm it is in the bundle; it reveals nothing to anyone else.
Sketches are not anonymous. With --sketch, the bundle also carries a skani sketch per genome.
It holds about 1 in 125 of the genome’s 15-mers with their positions and 1 in 1,000 of its 21-mers:
a sparse but real sample of the DNA. Names inside it are opaque (gtdbprep sketches the renamed
files), but the k-mers remain. prep prints this warning and writes it into the manifest. Don’t
send sketches for unpublished or sensitive genomes. The design describes a route that keeps the ANI
screen local (split GTDB-Tk, section 4).
How classify matches classify_wf
Section titled “How classify matches classify_wf”classify_wf runs the ANI screen first and never runs identify/align on genomes it assigns to
a species, so their msa_percent and translation_table are N/A. A plain gtdbtk classify
given a bundle that contains every genome fills those two fields in instead. That was the only
difference in the first round trip. gtdbprep classify therefore:
- runs GTDB-Tk’s own ANI screen (
ANIScreener.run_aniscreen, because GTDB-Tk 2.8 has noani_screensubcommand) on the renamed genomes. It runs under the Python that runsgtdbtk: the environment’spython3with--env, otherwise the interpreter named in the shebang of thegtdbtkscript onPATH(conda’s#!/bin/sh+execform included). The call refuses any GTDB-Tk version other than 2.8.0, since the API is internal, and refuses genome IDs it was not given; gtdbprep then checks every assigned ID is in the bundle’s manifest; - stages a copy of the bundle without the assigned genomes (rows removed from every TSV, records
from every MSA), which is what
classify_wf’s align folder looks like; - runs
gtdbtk classify --genome_dir <renamed> --align_dir <staged> [--full_tree]; - maps opaque IDs back to the original GTDB-Tk genome IDs (
replay/summary.original_names.tsv).
A placement server would do the same in steps 1 and 2, with the ANI screen fed by sketches.
All subprocesses run with LC_ALL=C. Under some host locales, Bioconda’s pplacer aborts in glibc’s
locale loader (_nl_intern_locale_data assertion), and the C locale changes no output.
Reference data
Section titled “Reference data”prep reads markers/, masks/ and metadata/. Their SHA-1 equals GTDB-Tk’s REF_HASHES
for R232 in GTDB’s 25 MB mockup archive (checked by prep on every run). GTDB-Tk also opens
taxonomy/gtdb_taxonomy.tsv, but only to refuse input genome IDs equal to a GTDB accession.
Opaque IDs (g + 16 hex digits) never are, so the mockup’s 5-line stub works. gtdbprep records
its hash and notes the REF_HASHES mismatch without stopping. These four folders total 77 MB. With
only them on disk, prep produced byte-identical alignment rows to a run against the whole
mockup. dblock add gtdbtk-r232-client && dblock fetch fetches exactly this subset and checks it
against REF_HASHES (see dblock); pass the path
dblock env prints to --data.
classify and verify need the full package (or the mockup for tests), as GTDB-Tk does.
Tested
Section titled “Tested”2026-11-10, GTDB-Tk 2.8.0 (Bioconda: Prodigal 2.6.3, HMMER 3.4, pplacer 1.1.alpha19, skani 0.4.0
for GTDB-Tk’s own ANI screen), skani 0.3.1 built from its tag for sketches, R232 mockup database
(md5 59512f60fa48010647977460bb855666, see GTDBTk#726),
4 threads on a 12-core Linux laptop. Genomes: GTDB’s test genome G021783475 (1.91 Mb, 138 contigs),
E. coli K-12 MG1655 (4.64 Mb) and P. aeruginosa PAO1 (6.26 Mb), gzipped.
Round trip (gtdbprep verify): 3/3 genomes identical to classify_wf. The merged
summary.tsv is identical line for line once rows are sorted. The test genome is assigned by the
ANI screen, and the other two are placed with pplacer (on the mockup’s 5-genome tree, so the
classifications themselves are not meaningful). The renamed genomes’ alignment rows are
byte-identical to classify_wf’s own user MSA. The whole verify (prep, classify from the
bundle, classify_wf) took 78 s wall.
| Bundle part | Bytes (3 genomes) | Per genome |
|---|---|---|
bac120.user_msa.fasta.gz |
7,499 | 2.5 KB |
| marker summaries (bac120 + ar53, plain text) | 5,782 | 1.9 KB |
| translation table, filtered/failed lists | 63 | <0.1 KB |
manifest.json |
5,794–6,343 | ~0.3 KB per genome entry + ~5 KB fixed |
| folder total, no sketches | 19,197–19,687 | 6.4–6.6 KB |
.tar.gz, no sketches |
11,201 | 3.7 KB |
| skani 0.3.1 sketches | 205,666 / 491,686 / 699,402 | 1.9 / 4.6 / 6.3 Mb genome |
Tests: 37 unit tests (renaming, GTDB-Tk genome IDs, gzip input, non-FASTA inputs, leak scan of
folders and tarballs, user/host/path needles, manifest round trip and tamper detection, SHA-256
vectors, GTDB-Tk’s sha1_dir order including symlinks, tar reproducibility, gzip header rewrite,
file modes, --out checks, summary comparison, dropping ANI-assigned genomes, the ANI-screen
interpreter, version and ID checks against a stand-in gtdbtk Python package). One of them runs
the whole prep against a stand-in gtdbtk shell script, with spaces in the input, data and
environment paths, a refused work folder with a space, two bad
inputs, and a local name planted where only the manifest would carry it. One integration test,
ignored by default, runs verify on GTDB’s test genome folder, copied into a folder whose name has
a space and joined by an empty input file. It checks
every genome is identical, that the empty input is reported as left out, that no original name is
in the .tar.gz, and that classify from the .tar.gz gives classify_wf’s rows:
GTDBPREP_ENV=/path/to/gtdbtk-env GTDBTK_DATA_PATH=/path/to/release232_mockup \GTDBPREP_TEST_GENOMES=/folder/with/test_genome_G021783475.fna.gz \GTDBPREP_SKANI=/path/to/skani-0.3.1 \cargo test --release -- --ignoredThe ANI species screen without the 85 GB database (ani)
Section titled “The ANI species screen without the 85 GB database (ani)”gtdbprep ani runs GTDB-Tk’s ANI screen (the first step of classify_wf, which gives species
names) on a laptop. It does not need the 85 GB skani database. How it works and what was
measured: The ANI species check on a laptop.
End to end, for GTDB R232:
# 1. Reference data, once: the screen index (2.1 GB download) and GTDB-Tk's taxonomy and# radii (the placement mirror's lite entry, 175 MB).dblock add gtdbtk-r232-ani-screen gtdbtk-r232-placement-litedblock fetchdblock env # prints both folders
# 2. GTDB-Tk 2.8.0 with skani pinned to 0.3.1 (the version the R232 database was made with)micromamba create -n gtdbtk -c conda-forge -c bioconda python=3.12 gtdbtk=2.8.0 skani=0.3.1
# 3. The screengtdbprep ani --genomes mags/ -x fna --work ani-work \ --index <store>/gtdbtk-r232-ani-screen/ani-screen-r232 \ --data <store>/gtdbtk-r232-placement-lite/release232 \ --env ~/micromamba/envs/gtdbtk --skani ~/micromamba/envs/gtdbtk/bin/skani \ --source /shared/gtdbtk/release232/skani/database/sketches.db--source says where sketches come from; there is no default. Without a copy of the skani
database, --source gtdb reads them from GTDB’s server (see below).
The results are GTDB-Tk’s own files, in ani-work/out/classify/ani_screen/:
gtdbtk.bac120.ani_summary.tsv, gtdbtk.ar53.ani_summary.tsv and
gtdbtk.ani_screen_unassigned_hits.tsv.gz.
GTDB-Tk’s usual summaries. ani-work/gtdbtk_data is then a complete GTDB-Tk data folder:
links to the placement-lite data plus a small skani database of the fetched references. With it
as GTDBTK_DATA_PATH, gtdbtk classify_wf runs its ANI screen on that database and writes the
usual classify/gtdbtk.bac120.summary.tsv (isotable reads it), in seconds for genomes the screen
assigns. Genomes it doesn’t assign go on to tree placement, which needs the
placement setup on a laptop.
From an AI agent. irit-mcp has this as the gtdbprep_ani tool, recorded and replayable like
the others, then gtdbtk_classify_wf with data_path: <work>/gtdbtk_data
(irit-mcp).
What it downloads per genome. Every reference that passes skani’s marker screen has to be
fetched, because GTDB-Tk’s files list all the close references, not only the species call. That
was 2 to 1,941 references per genome in the tests (B. subtilis 23, E. coli 420, P. aeruginosa
1,090, a 400 kb piece of E. coli 1,941), about 0.51 MB each. With the default source (GTDB’s
archive, --source gtdb, see below) each reference costs about 2.9 MB of download on average, so roughly:
| Genome | References | From GTDB’s archive (--source gtdb) |
From an unpacked sketches.db |
|---|---|---|---|
| B. subtilis 168 | 23 | about 70 MB | 12 MB |
| E. coli K-12 | 420 | about 1.2 GB | 0.2 GB |
| P. aeruginosa PAO1 | 1,090 | about 3.2 GB | 0.56 GB |
| 400 kb fragment | 1,941 | about 5.7 GB | 1.0 GB |
These are estimates from the measured average (9.3 GB for 3,166 references). Genomes from the
same genus share most references, and all runs share one cache (below): a second run of the
same genomes fetched nothing and took 18 s, and needs no --source.
If you only need the species, --species-only fetches far less. It fetches only the
references that could give the species call (see below). Measured with --source gtdb and an
empty cache: M. tuberculosis H37Rv 1 reference instead of 377 (2.9 MB instead of 1.23 GB),
E. coli K-12 18 instead of 420 (61 MB instead of 1.32 GB), B. subtilis 168 14 instead of 21.
The species call is the same; what can be missing are rows of the other two files (below). In
crowded genera the saving is smaller: five genomes from Streptococcus, Bacillus cereus,
Burkholderia, Pseudomonas and Klebsiella needed 542 references instead of 2,694 (1.68 GB
instead of 8.07 GB), with the same calls and the same related references.
--source gtdb is opt-in, at GTDB’s request. GTDB publishes the skani database only inside its 60.8 GB
.tar.gz. gtdbprep ani reads sketches out of that file on GTDB’s server with HTTP range
requests, through the seek index that comes with the screen index (a checkpoint every 4 MiB).
Each sketch costs the compressed bytes from the checkpoint before it, so about 5.7 times the
sketch itself, and the requests load GTDB’s server. GTDB’s maintainers asked that this be opt-in
and that fetched sketches be cached, never fetched twice
(GTDBTk#727); they don’t plan to publish
skani/database/ unpacked. So don’t run large batches this way: if you have the full package on
a shared disk, point --source at its sketches.db.
How it works:
- skani 0.3.1 sketches the genomes, with the database’s parameters.
- The genomes are screened against the screen index: every reference’s marker count and
32-bit fingerprints of its marker k-mers. This keeps every reference
skani searchwould compare in full (and perhaps a few more). - Only those references’ full sketches are fetched, by HTTP range requests, into the cache
(
--cache, default$XDG_CACHE_HOME/irit/gtdbprep-anior~/.cache/irit/gtdbprep-ani, shared by all work folders). Each is checked against a hash in the index, in the cache too. - They are written as a small skani database, and GTDB-Tk’s own
ANIScreenerruns against it. So the species rule (closest reference with AF >= 0.5, within its radius) and the output files are GTDB-Tk’s.
Options:
--sourcehas no default. It is needed only when some sketch is not in the cache.--source gtdb(opt-in) reads the sketches from GTDB’s owngtdbtk_r232_data.tar.gz, through the seek index in the screen index folder (gz.idx).--gtdb-urlpoints at another copy of the same archive (a URL or a local path).--source <path or URL of sketches.db>reads an unpackedsketches.db, local or hosted.--source '<url>/sketches.db.{part}' --part-size Nreads one split into numbered parts.--dataneedstaxonomy/,radii/andmetadata/(thegtdbtk-r232-placement-litemirror has them).--envis a GTDB-Tk 2.8.0 environment.--skanimust be the version the database was made with (0.3.1 for R232), or pass--allow-skani-version. Bioconda’s GTDB-Tk 2.8.0 brings skani 0.4.0 unless you pin it.--fetch-jobs N(default 4, at most 16): range requests in flight at once.1fetches one at a time, as earlier versions did. Failed requests (connection errors, a cut-off body, 408, 429 and 5xx) are retried up to 5 times with growing waits; a request that gets no answer for 60 s, or whose body arrives slower than 32 KiB/s, is cut off and retried. Each sketch is checked and written to the cache as soon as it arrives, so an interrupted run keeps what it fetched.--species-onlyfetches only the references that could give the species call. See below.--full-db <skani/database>is for testing: it runs the same GTDB-Tk call on a full database.
A genome skani makes no sketch of is left out of the screen, not fatal. skani skips assemblies
too short to sketch, such as a 407 bp assembly from 5× reads. gtdbprep prints
gtdbprep: warning: … it is not screened, screens the rest, and lists that genome with - for
its markers. If no genome can be sketched, the run fails.
<work>/screen.tsv lists every reference the screen kept for each genome: its shared marker
count, both marker counts, the count --species-only requires, and whether it was used.
--species-only. GTDB-Tk assigns a species from the closest reference with AF >= 0.5, and
only if that reference’s ANI reaches its own radius. No radius in R232 is below 95. A reference
that can’t reach ANI 95 can therefore never give the call. If it isn’t the closest, it doesn’t
matter. If it is the closest, every other reference is also below 95, so the genome is
unassigned either way. --species-only drops references whose marker count shows they can’t
reach that ANI, with a wide margin:
- it keeps a reference if the genome and the reference share at least the number of marker k-mers
expected at ANI 93 (2 points below the smallest radius, read from
radii/gtdb_radii.tsv) and AF 0.5, less 4 standard deviations (a binomial count). That is 10.9% of the smaller marker set (356 of 4,000 markers), against 0.9% in skani’s own screen; - on six test genomes (882 references screened in), every reference with ANI >= 95 and AF >= 0.5
was kept, the closest one to the limit with 4.7 times the shared markers required, and GTDB-Tk’s
ani_summary.tsvcame out byte-identical to the full run’s for all five assigned genomes.
What you lose: gtdbtk.ani_screen_unassigned_hits.tsv.gz (every skani hit of genomes the screen
does not assign) lists only the references that were fetched. For a test genome with no species
within reach, that was 2 rows instead of 14. GTDB-Tk’s classify reads this file for its notes
on genomes placed in the tree (closest reference, near misses), so those notes can differ too;
the placement and the classification itself don’t use it. The “other related references” column
of ani_summary.tsv lists the 50 closest references with AF >= 0.5, so it loses a row only if a
dropped reference had AF >= 0.5. No related reference was lost on any of the ten assigned test genomes
(for one, two tied references came out in the other order); on the five from crowded genera, every
dropped reference had AF below 0.5 (the highest ANI among them was 88.24). Measurements: The ANI species check on a laptop.
It prints, per genome, the number of marker k-mers and of references screened in, and the bytes fetched. Peak memory was 1.9 GB for 9 genomes (3,166 references); GTDB-Tk’s own screen on the full database took 14.7 GB.
Making the screen index (maintainers; needs the full database once):
gtdbprep ani-index build --db release232/skani/database --out ani-screen-r232 --release r232curl -s "$URL" | gtdbprep ani-index gz-scan --input - --out scan --url "$URL"gtdbprep ani-index gz-attach --index ani-screen-r232 --scan scan --every 4gz-scan reads the archive once (it can share one download with tar -x) and keeps a checkpoint
every --spacing bytes (default 1 MiB) of sketches.db. gz-attach --every N keeps every N-th
checkpoint: a smaller index, but more data downloaded per sketch.
Privacy. Nothing about the genomes leaves the laptop. The requests name byte ranges of the public reference database, so the server can see which references were fetched, and so roughly which taxa the genomes belong to.
Tree placement on the placement-only data (place)
Section titled “Tree placement on the placement-only data (place)”Genomes the ANI screen doesn’t assign to a species need tree placement. gtdbtk classify_wf can’t
do that on the gtdbtk-r232-placement-lite data: it aligns genomes against GTDB’s reference MSA
(7.9 GB), which that mirror leaves out. Placement itself doesn’t need the MSA. gtdbprep place
runs the route that does work: GTDB-Tk’s identify, align --skip_gtdb_refs, then classify
without the ANI screen, with gPlace instead of pplacer, under
a memory cap. Every step is GTDB-Tk’s (GTDB-Tk, Chaumeil
et al.), and the placements are gPlace’s (by GTDB-Tk’s author; identical to pplacer’s on 59
genomes, gPlace vs pplacer). gtdbprep only arranges the calls and
checks them. Cite GTDB, GTDB-Tk and gPlace for results.
gtdbprep place --genomes bins/ -x fa --work place-work \ --data <store>/gtdbtk-r232-placement-lite/release232 \ --env ~/micromamba/envs/gtdbtk --cpus 6 --max-memory 6G \ --skip-assigned ani-work # optional: the --work folder of `gtdbprep ani`What it does:
- Links the input files into
<work>/genomes/, since GTDB-Tk refuses genome paths with spaces, and lists them with GTDB-Tk’s own rules, so genome IDs match the onesanireports. With--skip-assigned, it reads<ani work>/assigned.jsonand the screen’sani_summary.tsvfiles, and unlinks the genomes the screen assigned: they need no placement. gtdbtk identify, thengtdbtk align --skip_gtdb_refs.gtdbtk classify --pplacer_cpus N, run through GTDB-Tk’s ownmain()after settingoptions.skip_ani_screen = True. That is the flagclassify_wfsets after its own ANI step; GTDB-Tk 2.8 has no command-line option for it, and the placement-only data has no skani database. The call refuses any GTDB-Tk version other than 2.8.0.- gPlace as the placer. GTDB-Tk has no option for choosing the placer
(GTDBTk#728). It runs
pplacer -m WAG -j N -c REFPKG -o OUT.json MSAby name, soplacewrites applacerscript into<work>/bin/and puts it first onPATH. The script runsgplace place --max_memory <cap>with the same arguments and logs each call to<work>/logs/gplace-calls.log. Onlypplacer --versiongoes to the real pplacer (GTDB-Tk logs it). Two checks: beforeclassify, the Python that runs GTDB-Tk confirms thatpplacerresolves to that script; afterwards, everypplacer.*.jsonplacement file under<work>/classifymust be the output of a logged gPlace call. Otherwise the run fails.
--max-memory (default 6G) goes to gPlace’s --max_memory. gPlace then splits a tree into tiles
when the whole placement wouldn’t fit. At 6G every R232 tree peaked at about 5 GiB
(gPlace vs pplacer). With the
full bacterial tree (--full-tree true, not in placement-lite), don’t go below about 5.2G.
--pplacer-cpus (default 1) is gPlace’s -j. --gplace names the program if it isn’t the
environment’s or the one on PATH.
Outputs in --work:
| Path | What |
|---|---|
place.tsv |
one row per input genome (below) |
classify/ |
GTDB-Tk’s own output: gtdbtk.bac120.summary.tsv (and ar53), trees, logs, the placement files in classify/intermediate_results/pplacer/ |
identify/, align/ |
GTDB-Tk’s identify and align output |
place.json |
the run record: tool versions (GTDB-Tk, gPlace), options, counts, times, number of gPlace calls |
logs/ |
one log per command, and gplace-calls.log |
place.tsv columns: genome, classification, method, gtdbtk_method, note,
pplacer_taxonomy, red_value, msa_percent, translation_table, closest_reference,
closest_reference_ani, closest_reference_af, warnings. method is:
placement: placed in the tree; the rest of the row is GTDB-Tk’s summary row (gtdbtk_methodis itsclassification_method,closest_referenceitsclosest_placement_reference; with no ANI screen these areN/A);ani_screen: skipped becausegtdbprep aniassigned it; the classification and the closest reference (with its ANI and AF) are the screen’s;not_placed: GTDB-Tk returned a row but no placement, for exampleUnclassified Bacteriafor a genome with too little of the marker alignment (seewarnings);no_result: GTDB-Tk returned no row (seeidentify/gtdbtk.failed_genomes.tsv).
The command prints the same table, then counts and times. It exits 0 on success and 2 on any
failure, with the reason on stderr. If every genome was assigned by the ANI screen, it runs no
GTDB-Tk step and writes the table from the screen alone. The work folder must be new, empty, or
one an earlier place made (marked by place.json): a rerun replaces place’s own files there and
nothing else.
Installing gPlace. gPlace 0.1.1 is not on PyPI or Bioconda (they have 0.1.0, as of
2026-11-10), and place was tested only with 0.1.1. On the test laptop it was built from source
with pip into the GTDB-Tk environment, which needs a C compiler (system gcc 16.2.1 there):
micromamba create -n gtdbtk -c conda-forge -c bioconda python=3.12 gtdbtk=2.8.0 skani=0.3.1micromamba run -n gtdbtk pip install \ git+https://github.com/cmc-aau/gPlace@695a401e0b54cc1bde55a5e1498a0697899b15eamicromamba run -n gtdbtk gplace check_install # "Installation OK ... bit for bit"Tested (2026-11-14): see the run below.
place on the Crystal Geyser bins
Section titled “place on the Crystal Geyser bins”2026-11-14, GTDB-Tk 2.8.0 and gPlace 0.1.1 (695a401) in one micromamba environment,
gtdbtk-r232-placement-lite, on the 12-core laptop. The input was the 23 MetaBAT 2 bins from Crystal
Geyser (MAGs on a laptop), with --skip-assigned set to the
gtdbprep ani work folder from that run, --cpus 6, --max-memory 6G. The whole command ran in a
systemd scope with MemoryMax=8G and MemorySwapMax=0.
- 20 bins were skipped as ANI-assigned. The other three were placed, with 3 gPlace calls: the backbone, class-level tree 6 and class-level tree 1.
- Same classifications as the earlier run: bin.12
Unclassified Bacteria(3.6% of the MSA), bin.13g__Sideroxyarcus;s__, bin.16g__OLB19;s__. GTDB-Tk’sgtdbtk.bac120.summary.tsvwas byte-identical to that of the run that used a separate shim and launcher outside gtdbprep, and so were the placements in all three placement files. - Peak memory 5.60 GiB (the scope’s
memory.peak, 6,017,150,976 bytes), no OOM kill. gPlace tiled all three trees; its largest estimate was 5.7 GiB, for class-level tree 1. - Wall time 11.3 min: identify 17 s, align 3 s, classify 652 s. The machine was shared, with a load average of 20 to 27 on 12 cores the whole time. The earlier run of the same classify took 270 s at a load of about 1.7, so most of the difference is likely contention.
Known gaps
Section titled “Known gaps”- Mockup only. The round trip ran against GTDB’s mockup database (5 reference genomes, full tree). The real R232 package (split backbone and class-level trees) is untested: that needs the ~100 GB package on a large-memory machine (the phased plan, step 2).
- Three bacterial genomes. No archaea, no translation table 4, no genomes below 10% alignment, no Prodigal failures, no batch where every genome is ANI-assigned, no borderline bac/ar calls.
- The ANI screen still needs the genomes.
classifyscreens the renamed FASTA files locally, so it runs where the genomes and the skani database both are. (gtdbprep aniruns the screen on the laptop without the database;classifydoesn’t use it yet.) Sketches are produced and pinned to skani 0.3.1, but nothing consumes them yet: the server-side shim that maps sketch queries back to opaque IDs (design, section 4) is not built. --place_speciesis not supported (itsclassify_wfbehaviour differs and is untested).- The ANI-screen step calls GTDB-Tk’s internal Python API (
ANIScreener.run_aniscreen), which may change in a later GTDB-Tk release. The call is pinned to GTDB-Tk 2.8.0:classifyrefuses other versions until the API has been checked against them. - Placement-side provenance for
classify(pplacer/gPlace version, full-package hash) is not written next to the results yet; only the laptop side has a manifest.placerecords the GTDB-Tk and gPlace versions inplace.json, but no reference-data hash. placedoes not write a merged GTDB-Tk summary. GTDB-Tk’ssummary.tsvin<work>/classifyholds only the placed genomes; ANI-assigned genomes are merged inplace.tsv, not in GTDB-Tk’s format.placewas tested on bacteria only (no archaea, no--full-tree), and with gPlace 0.1.1 only.- Marker summaries are plain text inside a folder bundle (1.9 KB per genome); use
.tar.gzto send them compressed. - Linux and macOS only (random IDs come from
/dev/urandom; GTDB-Tk doesn’t run on Windows). - Tool sources. Bioconda installs skani 0.4.0 with GTDB-Tk 2.8.0;
--sketchrefuses anything but 0.3.1, which has to be built from its tag. - Reference download. The
gtdbtk-r232-cliententry checks GTDB’s mockup archive against its published MD5 (corrected on 2026-11-11, GTDBTk#726) and the extracted folders againstREF_HASHES.