Image Storage and Management#

MinerU uses a pluggable storage abstraction for all image I/O. Every component that writes extracted images receives a DataWriter instance and calls writer.write(path, bytes) — the same call works regardless of whether the destination is a local directory, an S3 bucket, or an HTTP endpoint.

Storage Backend Abstraction#

The core interface is defined in mineru/data/data_reader_writer/base.py. DataWriter exposes two methods:

• write(path, data: bytes) — abstract; implemented by each backend.
• write_string(path, data: str) — encodes the string to UTF-8 (ASCII fallback with replacement) and delegates to write().

DataReader mirrors the pattern with read_at(path, offset, limit) as the core abstract method.

The public API is exported from mineru/data/data_reader_writer/__init__.py 1. S3 variants are lazy-loaded — they are only imported (and boto3 required) when first accessed by name 2, so the default install path carries no S3 dependency.

Available Implementations#

HTTP I/O (HttpReader / HttpWriter) sits one layer below in mineru/data/io/http.py and supports GET reads and multipart-POST writes; partial-range reads are not supported for HTTP.

S3 / S3-Compatible Integration#

Single-bucket shortcut#

S3DataReader and S3DataWriter are convenience wrappers that accept a single (bucket, ak, sk, endpoint_url, addressing_style) tuple and delegate to MultiBucketS3DataReader/Writer with a one-element config list. The addressing_style defaults to 'auto'; 'path' and 'virtual' are also valid (matching boto3 docs).

Multi-bucket routing#

MultiS3Mixin underpins both multi-bucket classes. It accepts a default_prefix (format: bucket/optional/prefix) and a list of S3Config objects — one per bucket, bucket names must be unique 3. Clients are instantiated lazily per bucket and cached in _s3_clients_h 4.

At runtime, read_at / write inspect the path: if it starts with s3://, the bucket is parsed from the URI and the matching client is selected; otherwise the default bucket and prefix are used 5.

Retry logic#

The underlying S3Reader and S3Writer configure the boto3 client with:

retries={'max_attempts': 5, 'mode': 'standard'}

6 — standard mode applies exponential backoff automatically. There is no additional retry layer in MinerU itself.

Configuration#

S3 credentials per bucket are read from ~/mineru.json (or the path set by MINERU_TOOLS_CONFIG_JSON) via get_s3_config(bucket_name). The config file's bucket_info key maps bucket names to [ak, sk, endpoint] tuples; a [default] entry is used as a fallback 7.

Image Deduplication via SHA-256#

Both PDF and HTML/Office pipelines use SHA-256 hashing to derive a content-stable filename, avoiding duplicate writes.

PDF pipeline — cut_image() in pdf_image_tools.py:

1. Constructs a logical path from page number and bounding box coordinates.
2. Hashes that path string with str_sha256(img_path).
3. Writes the cropped JPEG as <sha256>.jpg via the injected image_writer 8.

HTML/Office pipeline — save_base64_image() in html_image_utils.py:

1. SHA-256 hashes the entire base64 data URI string to derive the filename 9.
2. Uses _write_image_once() to skip the write entirely if the same path was already written within the same writer's lifetime — a simple set-based guard 10.

str_sha256 is a thin hashlib wrapper in mineru/utils/hash_utils.py that UTF-8-encodes its input before hashing.

Implication: Two images that produce identical logical paths (same page + bbox, or identical base64 payloads) will map to the same filename and will be written at most once per writer session, reducing both storage I/O and final artifact size.

Key Source Files#