BlobToolKit Data Storage#

A BlobDir is a plain filesystem directory — not a single database or class — where each dataset field is stored as a separate JSON file and a central meta.json tracks the dataset structure. This design keeps fields independent, enables lazy loading, and lets multiple data sources (coverage tracks, BUSCO, taxonomy) accumulate in the same directory without conflict.

Key source files:

src/blobtools/lib/field.py — field type classes and serialization
src/blobtools/lib/dataset.py — Metadata class managing meta.json
src/blobtools/lib/fetch.py — loading fields from disk
src/blobtools/lib/add.py — writing fields to disk
src/blobtools/lib/file_io.py — JSON/YAML/gzip I/O utilities

File Layout#

Every field is written as {DIRECTORY}/{field_id}.json , and the dataset metadata is always written to {DIRECTORY}/meta.json . A typical BlobDir contains:

MyDataset/
  identifiers.json # sequence IDs
  gc.json # GC content (Variable)
  length.json # sequence lengths (Variable)
  ncount.json # N-base counts (Variable)
  bestsumorder_phylum.json # taxonomy category
  bestsumorder_kingdom.json
  bestsumorder_positions.json # raw hit data (MultiArray)
  cov.json # coverage (Variable)
  meta.json # dataset + field metadata

Files can optionally be gzip-compressed (.json.gz); fetch_field() tries both variants when loading. JSON files are written using ujson.dumps with indent=1 .

Field Types#

Five field types are defined in field.py:

Type	Class	Stores
`identifier`	`Identifier`	Sequence IDs (must be unique)
`variable`	`Variable`	Numeric values (int or float) with min/max range
`category`	`Category`	Categorical strings, compactly encoded
`array`	`Array`	Per-sequence arrays (e.g., BUSCO scores)
`multiarray`	`MultiArray`	Jagged arrays (e.g., BLAST hit lists)

Each field serializes itself via values_to_dict(), which produces a dict with values and, where applicable, keys, category_slot, and headers. This dict is what gets written to {field_id}.json.

When loading, fetch_field() reads the type key from meta.json and dispatches to the correct class via the TYPES factory dict . The field data dict from disk is passed directly to the class constructor, so the stored format must exactly match the constructor's keyword arguments.

Compact Category Encoding#

Category, Array, and MultiArray fields compress repeated string values using a key-index scheme. _collapse_values() replaces each unique string with an integer index and returns a deduplicated keys list:

Input: ['Arthropoda', 'Arthropoda', 'Chordata', 'Arthropoda']
Output: keys=['Arthropoda', 'Chordata'], values=[0, 0, 1, 0]

On disk this becomes:

{"keys": ["Arthropoda", "Chordata"], "values": [0, 0, 1, 0]}

_expand_values() reverses this on load. For Array and MultiArray fields, one column of each per-sequence record can be a category — the category_slot index identifies which column to compress . This is used, for example, to store hit taxon names compactly within the hit position arrays produced by BLAST parsing .

Metadata-Driven Structure#

meta.json is written by Metadata.to_dict() and is the source of truth for what fields exist and how to load them. It records each field's type, datatype, range, scale, and position in a parent–child hierarchy.

add_field(parents, **kwargs) nests a new field's metadata under its parent path. For example, taxonomy fields are all nested under a "taxonomy" parent node, and hit-score sub-fields are nested under their rank field. This hierarchy is reflected in the UI but does not affect the flat {field_id}.json layout on disk.

has_field() checks a cached _field_ids set so duplicate-field warnings are fast. When blobtools add runs, it always re-writes meta.json at the end to capture newly added fields .

fetch_metadata() loads meta.json (or .json.gz) at the start of any command, and fetch_field() then uses the loaded Metadata object to determine each field's type before reading its JSON file from disk.

Multiple Taxonomy Sources in One BlobDir#

Taxonomy fields follow the naming convention {prefix}_{rank}.json. The prefix is set by --taxrule rule[=prefix] ; if no alternate prefix is given, the rule name itself is used (default: bestsumorder). For each taxonomic rank, a separate Category field and several sub-fields are generated :

{prefix}_{rank}.json — Category field (e.g., bestsumorder_phylum.json)
{prefix}_{rank}_score.json — bitscore for the winning taxon
{prefix}_{rank}_cindex.json — category diversity index
{prefix}_{rank}_positions.json — per-chunk taxon assignments
{prefix}_positions.json — raw hit data (MultiArray)

Running blobtools add --hits multiple times with different prefixes (e.g., --taxrule bestsumorder=nt then --taxrule bestsumorder=uniprot) produces parallel field sets that coexist in the same BlobDir. All taxonomy fields share the "taxonomy" parent node in meta.json, so the viewer can enumerate all available sources. The default plot category is set to {prefix}_phylum and can be changed with --update-plot.