Libmagic Pattern Matching#
libmagic-rs evaluates magic rules against a file buffer in three steps: read a value at an offset, compare it against a pattern, and advance an anchor for child rules with relative offsets. Pattern-bearing types — string (flagged), search, and regex — each have distinct scan-window definitions, comparison semantics, and anchor-advance rules that interact in non-obvious ways.
Primary source locations:
| Concern | File |
|---|---|
| Pattern comparison and anchor advance | src/evaluator/types/string.rs — compare_string_with_flags, read_string, read_string_exact, read_string16 |
| Type dispatch | src/evaluator/types/mod.rs — flagged_string_bytes_consumed, read_string16_dispatch, bytes_consumed_with_pattern_bounded |
search/N scan window | src/evaluator/types/search.rs |
regex scan window | src/evaluator/types/regex.rs |
| Relative offset resolution | src/evaluator/offset/relative.rs |
| Known pitfalls and oracle measurements | GOTCHAS.md §6 |
Scan-Window vs. Render Bounds#
Scan window — how far the evaluator searches for a pattern.
Render bounds — how many bytes/characters from the matched position are returned for %s substitution.
These are independent . Both affect what child rules see; confusing them produces either false negatives (pattern not found) or malformed renders.
search/N: Scan window is exactly N bytes from the rule offset. N is mandatory and non-zero (NonZeroUsize); bare search and search/0 are parse errors . The anchor advances to match-end, not window-end.
regex / regex/N: Scan window is min(8192, remaining_buffer) by default (mirroring libmagic's FILE_REGEX_MAX). A /N modifier caps at N; a /lN modifier extends through the Nth line terminator, still capped at 8192 bytes . The /s flag on both regex and search advances the anchor to match-start instead of match-end.
Flagged string: No separate scan window. The max_length AST field, when present, caps the buffer slice passed to compare_string_with_flags as buffer[..offset + cap] — not pre-sliced from offset, to avoid double-offsetting .
Render truncation: GNU file caps every rendered %s field at 127 bytes (MAXstring - 1). libmagic-rs mirrors this via MAX_DESCRIPTION_FIELD_LEN = 127 in src/output/format.rs . The scan-window cap (max_string_length, default 8192) and the render cap (127) are separate values applied at different stages.
Relative Offset Anchor Semantics#
Relative offsets (&+N, &-N) in magic rule syntax resolve against last_match_end, the byte position immediately after the previous successful match :
resolved = last_match_end + delta (bounds-checked; underflow → EvaluationError)
last_match_end is advanced by bytes_consumed_with_pattern_bounded after each successful match. Advance rules by type:
| Type | Anchor advances to |
|---|---|
Fixed-width (byte, short, long, quad, float, double, date) | offset + byte_width |
string (unflagged) | First NUL position + 1, or window end if no NUL |
string (flagged, Equal/NotEqual) | Pattern's declared length (see /w section below) |
pstring | Prefix width + payload bytes |
regex | Match-end (or match-start with /s) |
search | Match-end (or match-start with /s) |
string16 (lestring16/bestring16, equality match) | Pattern's byte length (matches libmagic m->vallen) |
string16 (any-value x) | Decoded character count (see below) |
A child &0 rule following an unflagged string match lands after the NUL, not on it. For flagged string, flagged_string_bytes_consumed returns effective.len() (the /T-trimmed pattern byte count) and never adds a byte for an adjacent 0x00, matching moffset() in softmagic.c .
Offset Advancement and String Flags (/w, /W, /c, /C, /f, /T)#
/w and /W — whitespace flexibility#
/w (compact_optional_whitespace) allows zero or more whitespace bytes in the file at each whitespace position in the pattern. /W (compact_whitespace) requires at least one. Both flags cause the file cursor to advance by the actual bytes consumed from the file, which may differ from pattern.len().
Critical: the relative-offset anchor does NOT advance by the walked byte count. flagged_string_bytes_consumed always advances by the pattern's declared length (m->vallen in libmagic's softmagic.c), regardless of flags. This was corrected from an earlier implementation that mistakenly used the walked count .
Example: pattern "a b" (3 bytes) with /w, matched against file "a b" (6 bytes). The comparator walks 6 file bytes; the anchor advances by 3. A child &0 rule reads at last_match_end + 3, not last_match_end + 6.
Regression tests: tests/relative_offset_evaluation.rs and src/evaluator/engine/tests/string_flags_dispatch_tests.rs .
/c and /C — asymmetric case folding#
Per libmagic softmagic.c :
/c(ignore_lowercase): fires only when the pattern char is lowercase. The file byte istolower'd before comparison. Uppercase pattern bytes are literal./C(ignore_uppercase): fires only when the pattern char is uppercase. The file byte istoupper'd before comparison.
Pattern FoO with /c matches at the o position but requires F and O to be literal. This asymmetry is the common source of "why doesn't my pattern match?" reports.
/f — full-word boundary#
/f (full_word) adds a post-match check: the byte immediately after the match must be end-of-buffer or a non-word character. This check fires after all /w//W whitespace consumption .
/T — trim#
When /T is set, flagged_string_bytes_consumed trims ASCII whitespace from both ends of the pattern before computing the anchor advance. The comparator receives the trimmed slice .
lestring16 / bestring16 Behavior#
What read_string16 does#
read_string16 decodes 16-bit code units from the buffer as UCS-2:
- Selects byte order from endianness (LE/BE).
- Stops at a
0x0000NUL pair, at the buffer end, or at theSTRING16_MAX_UNITScap (8192 units). - Ignores a trailing odd byte (non-even buffer remainder).
- Substitutes
U+FFFDfor surrogate halves (D800–DFFF), since UCS-2 does not resolve surrogates.
Why not full Unicode: An earlier implementation that decoded fully to Unicode broke the sfnt name-table rule &(&-14.S-17) lestring16 x, which reads one byte into a UTF-16BE sequence intentionally — the odd alignment makes the low bytes of each pair the "kept" bytes. Full Unicode decoding of those odd-aligned bytes produced Braille/CJK mojibake (Apple Braille.ttf, SFNS.ttf). The current UCS-2 narrowing is intentional (dosu_note:82b2fec0).
Comparison: prefix test, not whole-field#
lestring16/bestring16 comparison follows the same prefix-test semantics as string: libmagic's file_strncmp compares only the first m->vallen bytes of the pattern. Before PR #511, rmagic compared the entire decoded field, which silently disabled OLE stream-name rules :
- Prefix rules for
Dgn~,PerfectOffice_,256_,96_,Catalog,Thumbnailstream names in OLE2 compound docs never fired. - The
!WksSSWorkBookNotEqualgate fired spuriously when the stream name continued beyond that pattern. windows:474/windows:216rules for NUL-free UTF-16 text were dead; a.regexport was classified asdata.
The fix: read_string16_dispatch passes the full decoded value for comparison patterns, and comparison is bounded to the pattern's byte length at the compare step, matching m->vallen .
Anchor advance for string16#
- Equality match: anchor advances by the pattern's byte length.
- Any-value (
x):any_value_string16_boundadvances by the decoded character count, not the raw source byte count (which would be 2×). For non-ASCII content: 127 CJK characters decoded from 254 source bytes advance the anchor by 127, not 254. - NUL-terminated equality match:
string16_bytes_consumedincludes the 2-byte NUL terminator in the raw byte count.
High-byte output divergence#
Value::Bytes renders as U+FFFD in descriptions (via String::from_utf8_lossy). A kept high byte such as 0xa9 (copyright sign) that libmagic prints raw will appear as U+FFFD in rmagic output. This is accepted divergence — do not Latin-1-decode high bytes to "fix" it (dosu_note:82b2fec0).
String Read Modes and Value Types#
read_typed_value_with_pattern selects between two readers based on (max_length, pattern) :
read_string_exact(buffer, offset, length)— reads exactlylengthbytes with no NUL truncation. Used for pattern-comparison rules. Required for patterns with embedded NULs (e.g.,0 string PNCIHISK\0 Norton Utilities disc image data).read_string(buffer, offset, max_length)— reads until the first NUL or end of buffer, capped bymax_length. Used for the any-value (x) operator and%srendering.
Picking the wrong mode is a silent-never-match bug: pattern comparison → read_string_exact; printable scan → read_string.
Value::Bytes vs. Value::String: Patterns with high bytes (e.g., gzip's \037\213, OS/2 INF's HSP\x01\x9b\x00) parse as Value::Bytes. Using from_utf8_lossy to convert these to Value::String inflates the byte length (U+FFFD is 3 bytes vs. 1 source byte) and changes the byte values, causing read_string_exact to read the wrong count and making the pattern never match .
offset == buffer.len() is not an overrun for string: read_string at EOF returns Value::String("") rather than BufferOverrun, matching GNU file behavior for truncated headers .
Common Pitfalls#
| Pitfall | Correct behavior | Source |
|---|---|---|
Using walked byte count as anchor advance for /w//W rules | Advance by pattern's declared length, not the walked count | |
Pre-slicing buffer from offset before passing to compare_string_with_flags | Pass buffer[..offset + cap]; the comparator slices from offset internally | |
Expecting anchor to skip past a trailing NUL for flagged string | Flagged strings never bump past an adjacent NUL; the anchor lands on it | |
Whole-field string16 comparison instead of prefix test | Comparison is bounded to pattern.len() bytes, matching m->vallen | |
Conflating string16 any-value anchor units with raw byte count | Any-value anchor uses decoded character count, not 2 × characters | |
Decoding high bytes in string16 values to Latin-1 | Value::Bytes renders as U+FFFD; accepted divergence — do not "fix" | |
Using from_utf8_lossy to convert Value::Bytes patterns to Value::String | Inflates byte length and corrupts values; use Value::Bytes directly | |
Reading bare &0 child after unflagged string as landing on the NUL | Anchor lands after the NUL (NUL is included in the consumed count) | |
Treating read_pattern_match returning Ok(Some("")) as a miss | It is a zero-width match (e.g., ^); only Ok(None) is a genuine miss |
Full reference: GOTCHAS.md §6 covers string-type pitfalls with oracle measurements, including read-mode selection (S6.4), case-flag asymmetry (S6.5), high-byte Value::Bytes handling (S6.7), and the anchor-advance correction (S6.8).