MinerU Doclib System

MinerU Doclib System#

Status: v4.x ("next") feature — not present in the stable v3.4.x release. The mineru/doclib/ module exists on the next development branch but is not part of the published mineru-3.x packages.

The doclib (document library) is MinerU's v4.x database-backed infrastructure for storing, indexing, and retrieving parsed documents. Rather than treating each parse as a one-shot file conversion, the doclib maintains a persistent, multi-tier content store that supports structured addressing, progressive reading, and full-text search across large document collections. It lives under mineru/doclib/ and is the central subsystem behind the mineru read, mineru search, mineru find, and mineru watch CLI commands introduced in the v4.x track .

Key components :

mineru/doclib/locators.py — URI regex and parsing for block locator strings
mineru/doclib/server.py — HTTP-level doclib service; hosts _build_read_plan_from_locator() and search()
mineru/doclib/services/parse_svc.py — Parse request routing and tier-aware caching
mineru/doclib/services/search_svc.py — Full-text search with tier filtering
mineru/doclib/core/db.py — Relational database schema (docs, files, parses, FTS tables)
mineru/doclib/core/fts.py — Full-text search index; replace() accepts tier: Tier | None
mineru/doclib/migrations/ — SQL migration scripts (e.g., 002_map_legacy_tiers.sql for tier renaming)

URI-Like Block Locator Strings#

Doclib uses a hierarchical, URI-style string to address any location within a parsed document :

doc:<short_id>/tier:<tier>/page:<N>[/block:<M>[/char:<offset>]]

Segment	Values	Notes
`doc:<short_id>`	hex short ID	assigned at parse time
`tier:<tier>`	`flash`, `medium`, `high`, `extra_high`	identifies which parse cache to read from
`page:<N>`	1-based integer	`page:last` is not yet supported
`block:<M>`	1-based integer	optional
`char:<offset>`	0-based integer	optional; requires `block` to be present

Example locators:

doc:a1b2c3d4/tier:medium/page:1 — first page of a medium-tier parse
doc:a1b2c3d4/tier:high/page:5/block:3/char:42 — character-level cursor inside a block

The regex in mineru/doclib/locators.py was updated in PR #5236 to accept extra_high and drop the legacy standard/pro tier names . The migration 002_map_legacy_tiers.sql back-fills existing doclib rows: standard → medium and pro → high .

Locators are used as content cursors in mineru read, as continuation markers in progressive reading, and in API/SDK call contracts. For text files (.txt, .md, .csv, .rst, .tex), a locator targeting a read operation returns a parse_not_required error — text files are indexed for FTS but do not produce parse rows .

CLI Architecture: v3.x vs. v4.x#

v3.x (current stable — 3.4.x)#

The production CLI lives in mineru/cli/ as a flat set of modules :

mineru/cli/client.py — the main mineru entry point (-p, -o, -b, --effort, etc.)
mineru/cli/fast_api.py — mineru-api server (FastAPI, async task queue)
mineru/cli/router.py — mineru-router multi-upstream dispatcher

Backends are hybrid-engine (default), hybrid-http-client, plus legacy aliases for pipeline (routed to hybrid-engine --effort low) . Parse output is written directly to a user-specified output directory — no persistent library.

v4.x ("next" track — in development)#

The v4.x CLI reorganizes into a mineru/cli/commands/ subpackage alongside a mineru/kit/ entrypoint . New commands added in this track:

Command	Purpose
`mineru parse`	Parse and ingest a document into doclib
`mineru read <locator>`	Retrieve page/block content by locator
`mineru search <query>`	Full-text search across the doclib
`mineru find <filename>`	Filename-based discovery; FTS wildcards required for prefix match
`mineru watch`	Monitor a directory for automatic ingestion
`mineru list files`	List all doclib-tracked documents
`mineru invalidate <locator>`	Evict a cached parse result

The mineru-kit api-server command serves a single tier and is composable — run multiple instances to serve medium and high simultaneously, with doclib routing between them . The old mineru/cli/ modules are relocated to mineru/cli_old/ during the transition .

Tier Naming Across Versions#

v3.x name	v4.x name	Description
`pipeline`	`hybrid-engine --effort low`	Local, CPU-friendly, no VLM
`standard`	`medium`	Local hybrid, default quality
`pro`	`high`	VLM-assisted, higher accuracy
—	`extra_high`	Full VLM (two-pass), max quality
`flash`	`flash`	Fast indexing/watch tier (not user-quality default)

Doclib Database and Tier Semantics#

The doclib persists state in a relational database managed by mineru/doclib/core/db.py. Core tables include docs, files, parses (one row per doc × tier), and fts_contents (full-text index). The FTS layer (mineru/doclib/core/fts.py) stores a tier value that may be Tier | None — None indicates a text file indexed directly, not through a parse job .

Tier selection at read time: When tier or min_tier is omitted from a mineru search call, results at all tiers — including tier=null text entries — are returned. Explicit tier filters exclude null-tier text results .

Default parse-server selection: When a client omits tier, the local server picks the highest available non-flash tier (high > medium). If neither is available, the request fails with quality_tier_unavailable rather than silently falling back to flash. The official remote API (mineru.net/api) currently serves only high .

Known operational issues (as of July 2026):

page:last symbolic addressing not yet implemented — only numeric page indices are accepted
mineru find requires explicit FTS wildcards (e.g. bench*) for prefix matching
mineru read --output exports Markdown but does not co-export the images/ directory; images can be fetched separately via mineru read <block-locator> --format image
mineru watch rescan may fail to bind pre-existing files to a new watch ID