OpenDAL Read Validation (CompleteLayer / CompleteReader)#

Overview#

OpenDAL enforces exact-size read guarantees on bounded range reads through CompleteLayer and its inner CompleteReader wrapper. When a caller requests a specific byte range (e.g., bytes=1024-2047), OpenDAL validates that the underlying storage backend returned exactly the requested number of bytes — no more, no less. Short reads that stop before reaching the requested size (e.g., due to a truncated file) are treated as errors, not silent data loss.

CompleteLayer is an internal, mandatory layer — not a user-configurable option. Every Operator automatically has it applied once and only once during construction. 1

How It's Applied#

OperatorBuilder::new() wires up three mandatory layers in order: ErrorContextLayer, CompleteLayer, and CorrectnessCheckLayer. 2 CompleteLayer is never exposed to external users and cannot be bypassed.

When the Operator calls read(), CompleteAccessor extracts the size from OpRead's BytesRange (the Option<u64> second field), then wraps the backend reader in a CompleteReader initialized with that size. 3

BytesRange and Bounded vs. Unbounded Reads#

BytesRange is a (offset: u64, size: Option<u64>) tuple: 4

• Bounded — size = Some(n): read exactly n bytes starting at offset. Maps to HTTP Range: bytes=<offset>-<offset+n-1>. 5
• Unbounded — size = None: read to end of content. Maps to Range: bytes=<offset>-. 5

CompleteReader only enforces size validation when size is Some. An unbounded read (size = None) has no byte-count check. 6

CompleteReader: The Validation Mechanism#

CompleteReader<R> wraps the inner reader and carries three fields: the inner reader R, an optional size: Option<u64> (the expected byte count), and a running read: u64 counter.

The read loop accumulates bytes into self.read. When the inner reader returns an empty buffer (EOF signal), check() is called immediately:

Both "too little" and "too much" errors include expect and actual context fields for diagnostics. 6

CompleteWriter: Analogous Write Validation#

CompleteWriter applies the same pattern on writes: it tracks bytes written via self.size, then on close() calls check(content_length) against the content_length reported by the underlying writer. Append-mode and zero-length writes skip this check. In debug builds, dropping an unclosed writer emits a log::warn!. 7

Other Completions in CompleteLayer#

Read/write size validation is one responsibility of CompleteLayer. The same layer also:

• Stat Completion — simulates stat on directories via list when the backend lacks native dir-stat support. 8
• List Completion — wraps listers with FlatLister or PrefixLister to emulate list_with_recursive and prefix filtering for backends that don't support them natively. 9
• create_dir Completion — synthesizes create_dir via an empty write when the backend supports write_can_empty and list. 10

Key Source Files#