Architecture & Design Internals#

Introduction#

This document is the definitive internal engineering reference for maintainers and AI agents performing non-trivial feature work on Stringy. It synthesises information from the handbook docs (docs/src/), the Kiro specification suite (.kiro/specs/stringy-binary-analyzer/), and the implementation PR history into a single, citable authority.

What is Stringy?#

Stringy is a format-aware alternative to the standard strings command 1. Rather than dumping every run of printable bytes, it leverages deep knowledge of executable file structures to extract only the strings that are genuinely part of the binary's data, producing ranked, semantically-tagged output that analysts can act on immediately.

The six core design principles are 2:

Advantages over strings#

• Eliminates noise: stops dumping padding, tables, and interleaved garbage 3
• UTF-16 support: surfaces wide strings crucial for PE analysis
• Actionable buckets: categorised results (URLs, keys, registry paths) are surfaced first
• Provenance tracking: offset and section info is preserved for pivoting to other tools
• YARA integration: feeds only high-signal candidates into YARA rules

Known Limitations#

• No reachability proof: without real cross-references, any "referenced" flag is opportunistic 4
• Packed binaries will appear sparse until unpacked 5
• Go and .NET have richer metadata — support is possible but deliberately kept in scope for later milestones

Source Files#

All claims in this document are backed by specific source files. For direct reading:

Implementation Status Legend#

Throughout this document, component status is indicated as:

• ✅ Implemented — shipped and tested
• 🚧 Framework Ready / Types Defined — scaffolding in place, logic pending
• 📋 Planned — not yet started

Overall Architecture and Pipeline Stages#

Source: docs/src/architecture.md, .kiro/specs/stringy-binary-analyzer/design.md

Stringy is built as a modular Rust library with a clear separation of concerns 6. Binary data flows linearly through a processing pipeline, with each stage enriching the data and passing it to the next.

High-Level Pipeline#

Binary File → Format Detection → Container Parsing → String Extraction
           → Encoding Detection → Deduplication → Classification
           → Symbol Processing → Ranking → Output

The formal specification describes the same flow as a graph 7:

Input File → Format Detection → Container Parser → Targeted String Extraction
           → Encoding Detection & Conversion → Deduplication & Canonicalization
           → Semantic Classification → Symbol Processing → Ranking Algorithm
           → Output Formatter → Results

Core Design Principles#

Every architectural decision in Stringy flows from five explicit principles 8:

1. Format-First Approach — every extraction decision is informed by file format knowledge; there is no generic "scan all bytes" fallback
2. Lazy Evaluation — optional features (DWARF, disassembly) are computed only when requested, keeping the common path fast
3. Memory Efficiency — memory mapping (memmap2) is used for large files; streaming processing is used where possible
4. Extensible Design — new file formats are added through feature gates, maintaining a lightweight core binary
5. Performance-Conscious — compiled regexes are cached, allocations in hot paths are minimised

Core Modules#

main.rs
├── lib.rs ← public API surface
├── types.rs ← core data structures
├── container/ ← ✅ Implemented
│ ├── mod.rs (format detection)
│ ├── elf.rs (ELF parser)
│ ├── pe.rs (PE parser)
│ └── macho.rs (Mach-O parser)
├── extraction/ ← 🚧 Framework Ready
│ ├── mod.rs (extraction traits)
│ ├── ascii.rs (ASCII/UTF-8)
│ ├── utf16.rs (UTF-16LE/BE)
│ └── dedup.rs (deduplication)
├── classification/ ← 🚧 Types Defined
│ ├── mod.rs (classification framework)
│ ├── semantic.rs (pattern matching)
│ ├── symbols.rs (symbol processing)
│ └── ranking.rs (scoring algorithm)
└── output/ ← 🚧 Interfaces Defined
    ├── mod.rs (output traits)
    ├── json.rs (JSONL format)
    ├── human.rs (table format)
    └── yara.rs (YARA format)

9

Module Responsibilities#

1. Container Module — ✅ Implemented#

Handles binary format detection and parsing using the goblin crate 10.

pub trait ContainerParser {
    fn detect(data: &[u8]) -> bool
    where
        Self: Sized;
    fn parse(&self, data: &[u8]) -> Result<ContainerInfo>;
}

11

The format detection pipeline is 12:

1. detect_format() uses goblin::Object::parse() to identify the format
2. create_parser() returns the appropriate Box<dyn ContainerParser>
3. The parser extracts sections, imports, and exports with full metadata

The parsers assign section weights based on the likelihood of containing meaningful strings 13:

// ELF section weights
".rodata" | ".rodata.str1.*" => 10.0 // Highest priority
".comment" | ".note.*" => 9.0 // Build info — very likely strings
".data.rel.ro" => 7.0 // Read-only after relocation
".data" => 5.0 // Writable data
".text" => 1.0 // Code sections (lowest priority)

14

2. Extraction Module — 🚧 Framework Ready#

Implements encoding-aware string extraction with configurable parameters 15:

• ASCII/UTF-8: scans for printable character sequences with noise filtering
• UTF-16: detects little-endian and big-endian wide strings with confidence scoring
• Deduplication: canonicalises strings while preserving complete metadata
• Section-Aware: uses container parser weights to prioritise extraction areas

3. Classification Module — 🚧 Types Defined#

Applies semantic analysis to extracted strings 16:

• Pattern Matching: regex to identify URLs, IPs, paths, GUIDs, etc.
• Symbol Processing: demangling for Rust and C++ symbols
• Context Analysis: section context and source type inform classification
• Extensible Tags: 15+ semantic categories

4. Ranking Module — 🚧 Algorithm Designed#

Implements the scoring algorithm 17:

Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty

See § Ranking and Scoring Model for complete scoring detail.

5. Output Module — 🚧 Interfaces Defined#

Formats results for different use cases 18:

• Human-readable: sorted tables with score, offset, section, tags
• JSONL: complete structured data with all metadata fields
• YARA: properly escaped strings with confidence grouping

Core Data Structures#

The central data type that flows through the entire pipeline is FoundString 19:

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct FoundString {
    pub text: String,
    pub encoding: Encoding,
    pub offset: u64,
    pub rva: Option<u64>,
    pub section: Option<String>,
    pub length: u32,
    pub tags: Vec<Tag>,
    pub score: i32,
    pub source: StringSource,
    // Debug-mode fields (populated when --debug is active)
    pub original_text: Option<String>, // pre-demangle text
    pub section_weight: Option<i32>,
    pub semantic_boost: Option<i32>,
    pub noise_penalty: Option<i32>,
}

The #[non_exhaustive] attribute is applied to this struct to allow future field additions without breaking pattern-match exhaustiveness in downstream code 20.

Data Flow Code Sketch#

// 1. Format detection and container parsing
let format = detect_format(&data);
let parser = create_parser(format)?;
let container_info = parser.parse(&data)?;

// 2. String extraction from prioritised sections
let mut all_strings = Vec::new();
for section in container_info.sections.iter().filter(|s| s.weight > 5.0) {
    let strings = extract_strings(&data, &section, &config)?;
    all_strings.extend(strings);
}
all_strings.extend(extract_symbol_strings(&container_info));

// 3. Deduplication
let unique_strings = deduplicate(all_strings);

// 4. Classification
for string in &mut unique_strings {
    let context = StringContext {
        section_type: string.section_type,
        source: string.source,
        encoding: string.encoding,
    };
    string.tags = classify_string(&string.text, &context);
    string.score = calculate_score(&string, &context);
}

// 5. Sort and output
unique_strings.sort_by_key(|s| std::cmp::Reverse(s.score));
let filtered = apply_filters(&unique_strings, &config);
format_output(&filtered, &config.format)

21

External Dependencies#

22 23

String Extraction#

Source: docs/src/string-extraction.md

Stringy's extraction engine is designed to find meaningful strings while aggressively suppressing noise and false positives. The approach is encoding-aware, section-aware, and fully configurable.

Extraction Sub-Pipeline#

Binary Data → Section Analysis → Encoding Detection → String Scanning
           → Noise Filtering → Deduplication → Classification

24

ASCII Extraction#

ASCII is the foundational encoding in most binaries 25. The extractor scans for runs of printable characters in the range 0x20–0x7E, applying a configurable minimum length threshold (default: 4 characters).

fn extract_ascii_strings(data: &[u8], min_len: usize) -> Vec<RawString> {
    let mut strings = Vec::new();
    let mut current_string = Vec::new();
    let mut start_offset = 0;

    for (i, &byte) in data.iter().enumerate() {
        if is_printable_ascii(byte) {
            if current_string.is_empty() { start_offset = i; }
            current_string.push(byte);
        } else {
            if current_string.len() >= min_len {
                strings.push(RawString {
                    data: current_string.clone(),
                    offset: start_offset,
                    encoding: Encoding::Ascii,
                });
            }
            current_string.clear();
        }
    }
    strings
}

26

Configuration:

let config = AsciiExtractionConfig::default(); // min_length: 4
let config = AsciiExtractionConfig::new(8); // custom min
config.max_length = Some(256); // add upper bound

27

UTF-8 Extraction#

UTF-8 extraction builds on ASCII extraction and handles multi-byte characters, providing a superset of ASCII coverage 28.

UTF-16 Extraction#

UTF-16 is critical for Windows PE binaries, where the Windows API ecosystem heavily favours wide strings 29. Stringy supports both byte orders with advanced false-positive prevention.

UTF-16LE (Little-Endian) ✅#

Most common on Windows platforms. Detection heuristics 30:

• Even-length byte sequences (2-byte alignment required)
• Low byte is printable; high byte is mostly zero
• Null termination pattern is 0x00 0x00
• Advanced multi-heuristic confidence scoring

UTF-16BE (Big-Endian) ✅#

Found in Java .class files, network protocols, some cross-platform binaries 31:

• Same structure as LE, but byte order is reversed
• High byte is printable; low byte is mostly zero

Automatic Byte-Order Detection ✅#

ByteOrder::Auto mode simultaneously scans for both LE and BE strings, avoids duplicates, and correctly identifies the encoding of each result 32.

UTF-16 Confidence Scoring#

UTF-16 is prone to false positives because binary data with null bytes can look like wide strings. The confidence formula combines five heuristics 33:

confidence = (valid_unicode_weight × valid_ratio)
           + (printable_weight × printable_ratio)
           + (ascii_weight × ascii_ratio)
           - (null_pattern_penalty)
           - (invalid_range_penalty)

The five contributing factors:

1. Valid Unicode range check — code points validated against U+0020–U+D7FF, U+E000–U+FFFD, U+10000–U+10FFFF; private-use areas and invalid surrogates are penalised
2. Printable character ratio — including common Unicode ranges
3. ASCII ratio boost — confidence boosted when >50% of characters are printable ASCII
4. Null pattern detection — flags suspicious patterns: >30% nulls, nulls at fixed intervals (every 2nd, 4th, 8th position)
5. Byte order consistency — verifies byte order is consistent throughout the string (for Auto mode)

Example confidence levels 34:

• "Microsoft Corporation" → High (>90% printable, valid Unicode, no null patterns)
• "Test123" → Medium (>70% printable, valid Unicode)
• Binary table [0x01, 0x00, 0x02, 0x00, ...] → Low (excessive nulls, regular pattern)

Practical recommendations 35:

UTF-16 Configuration#

use stringy::extraction::utf16::{Utf16ExtractionConfig, ByteOrder};

// Windows PE — LE only, higher confidence bar
let config = Utf16ExtractionConfig {
    byte_order: ByteOrder::LE,
    min_length: 3,
    confidence_threshold: 0.6,
    ..Default::default()
};

// Unknown binary — auto-detect both orders
let config = Utf16ExtractionConfig {
    byte_order: ByteOrder::Auto,
    ..Default::default() // min_length: 3, confidence_threshold: 0.5
};

36

Noise Filtering System#

Stringy implements a multi-layered heuristic filtering system that runs after extraction to score each string's legitimacy on a 0.0–1.0 confidence scale 37. The system adds less than 10% overhead compared to unfiltered extraction 38.

The six independent filter layers are combined using configurable weights:

1. Character Distribution Filter#

Detects abnormal character frequency distributions 39:

• 80% punctuation → confidence 0.2

• 90% same character → confidence 0.1 (e.g. "AAAAAAA")

• 70% non-alphanumeric → confidence 0.3

• Reasonable distribution → confidence 1.0

2. Entropy Filter (Shannon Entropy)#

Entropy is measured in bits-per-byte 40:

3. Linguistic Pattern Filter#

Analyses text for word-like properties 41:

• Vowel-to-consonant ratio in acceptable range (0.2–0.8 for English)
• Presence of common English bigrams: th, he, in, er, an, re, on, at, en, nd
• Graceful handling of non-English strings (no over-penalisation)

4. Length Filter#

Length-based confidence penalties 42:

• 4–100 characters → confidence 1.0

• 200 characters → confidence 0.3 (likely table data)

• < 4 characters in low-weight sections → confidence 0.5

5. Repetition Filter#

Identifies repetitive patterns 43:

• Repeated characters (e.g. "AAAA", "0000") → confidence 0.1
• Repeated substrings (e.g. "abcabcabc") → confidence 0.2
• Normal string → confidence 1.0

6. Context-Aware Filter#

Adjusts confidence based on where a string was found 44:

Filter Weights and Configuration#

All six filters are combined using a weighted average. The default weights sum to 1.0 45:

pub struct FilterWeights {
    pub entropy_weight: f32, // Default: 0.25
    pub char_distribution_weight: f32, // Default: 0.20
    pub linguistic_weight: f32, // Default: 0.20
    pub length_weight: f32, // Default: 0.15
    pub repetition_weight: f32, // Default: 0.10
    pub context_weight: f32, // Default: 0.10
}

Confidence thresholds:

• ≥ 1.0 → maximum confidence (imports, exports, resource strings)
• 0.7–0.9 → high confidence (likely legitimate)
• 0.5–0.7 → moderate confidence (may need review)
• < 0.5 → filtered out by default

Note: extraction confidence (0.0–1.0) is distinct from the final ranking score (0–100). Confidence represents the noise-filtering assessment; the score is a composite that includes section weight and semantic tags and is described fully in § Ranking and Scoring Model.

Section-Aware Extraction#

Different sections use different extraction strategies based on their expected string density 46:

PE Resource Extraction#

The .rsrc section is handled via structured parsing rather than raw byte scanning 47:

fn extract_pe_resources(pe: &PE, data: &[u8]) -> Vec<RawString> {
    let mut strings = Vec::new();
    if let Some(version_info) = extract_version_info(pe, data) {
        strings.extend(version_info); // VERSIONINFO key-value pairs
    }
    if let Some(string_tables) = extract_string_tables(pe, data) {
        strings.extend(string_tables); // Localised UI strings
    }
    strings
}

Deduplication Strategy#

After extraction, strings are deduplicated while preserving all occurrence metadata 48.

Canonicalisation steps:

1. Normalise whitespace (tabs/newlines → spaces)
2. Trim leading/trailing whitespace
3. Preserve original case (case-insensitive comparison is not used)
4. Normalise to UTF-8 for comparison

Data model — each unique string tracks all occurrences:

struct DeduplicatedString {
    canonical_text: String,
    occurrences: Vec<StringOccurrence>,
    primary_encoding: Encoding,
    best_section: Option<String>,
}

struct StringOccurrence {
    offset: u64,
    section: Option<String>,
    encoding: Encoding,
    length: u32,
}

Algorithm — HashMap-based O(1) lookup 49:

fn deduplicate_strings(strings: Vec<RawString>) -> Vec<DeduplicatedString> {
    let mut map: HashMap<String, DeduplicatedString> = HashMap::new();
    for string in strings {
        let canonical = canonicalize(&string.text);
        map.entry(canonical.clone())
           .or_insert_with(|| DeduplicatedString::new(canonical))
           .add_occurrence(string);
    }
    map.into_values().collect()
}

Design Note: An early implementation used O(n²) deduplication in Auto byte-order mode. This was replaced with the HashMap-based approach in a correctness/performance cleanup pass 50.

Performance Optimisations#

The extraction layer uses three performance strategies 51:

1. Memory mapping via memmap2 for files exceeding the configured threshold (avoids loading the entire file into heap memory)
2. Parallel section processing via rayon::par_iter() — each section is an independent unit of work
3. Regex caching via lazy_static! — URL, GUID, and other patterns are compiled once and reused across all strings

Classification System#

Source: docs/src/classification.md

Stringy's classification system applies semantic analysis to extracted strings, tagging them with machine-readable labels that help analysts quickly identify the most operationally relevant data.

Classification Pipeline#

Raw String → Pattern Matching → Context Analysis → Tag Assignment → Confidence Scoring

52

Semantic Tag Taxonomy#

Stringy defines 15+ semantic tag categories 53:

Pattern Definitions#

Network Indicators#

URLs 54:

• Pattern: https?://[^\s]+
• Confidence factors: valid TLD, path structure, parameter format
• https:// prefix adds +0.3 confidence; valid TLD adds +0.2

Domain Names 55:

• Pattern: [a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
• Validated against TLD list and DNS format compliance
• URL classifier takes priority when both patterns match (domains inside URLs are not double-tagged)

IP Addresses 56:

• IPv4 pattern: \b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b with range validation
• IPv6 pattern: \b(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}\b with bracket/port notation support
• Reserved address detection is included in validation

File System Indicators#

File Paths 57:

• POSIX pattern: /[^\0\n\r]*
• Windows pattern: [A-Za-z]:\\[^\0\n\r]*
• UNC path support also included
• Context boosts confidence when the surrounding strings suggest a filesystem context

Registry Paths 58:

• Pattern: HKEY_[A-Z_]+\\[^\0\n\r]*
• High security relevance as a persistence mechanism indicator

Identifiers#

GUIDs/UUIDs 59:

• Pattern: \{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\}
• UUID version checking is performed
• COM class identifiers are a primary use case

Email Addresses 60:

• Pattern: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
• RFC-compliant with domain validation

Code Artifacts#

Format Strings 61:

• Pattern: %[sdxo]|%\d+[sdxo]|\{\d+\}
• Covers both printf-style and .NET-style format specifiers

Base64 Data 62:

• Pattern: [A-Za-z0-9+/]{20,}={0,2}
• Validated for length divisibility and correct padding
• High variability in security relevance (may be encoded payloads)

User Agents 63:

• Pattern: Mozilla/[0-9.]+|Chrome/[0-9.]+|Safari/[0-9.]+

Implementation: Pattern Matching Engine#

All patterns are compiled once at startup using once_cell::sync::Lazy (or lazy_static!) and reused across all classification calls to avoid expensive recompilation 64:

pub struct SemanticClassifier {
    url_regex: Regex,
    domain_regex: Regex,
    ipv4_regex: Regex,
    ipv6_regex: Regex,
    guid_regex: Regex,
    email_regex: Regex,
    format_regex: Regex,
    base64_regex: Regex,
}

impl SemanticClassifier {
    pub fn classify(&self, text: &str, context: &StringContext) -> Vec<Tag> {
        let mut tags = Vec::new();
        if self.url_regex.is_match(text) { tags.push(Tag::Url); }
        if self.domain_regex.is_match(text) && !tags.contains(&Tag::Url) {
            tags.push(Tag::Domain); // URL takes priority over Domain
        }
        if self.is_file_path(text) { tags.push(Tag::FilePath); }
        if self.is_registry_path(text) { tags.push(Tag::RegistryPath); }
        // ... (other patterns)
        tags
    }
}

65

Context-Aware Classification#

Classification is not purely pattern-based. The StringContext struct carries section metadata that allows classifiers to adjust their output 66:

pub struct StringContext {
    pub section_type: SectionType,
    pub section_name: Option<String>,
    pub surrounding_strings: Vec<String>,
    pub binary_format: BinaryFormat,
    pub encoding: Encoding,
}

Context-specific behaviour:

• Strings in SectionType::Resources have version string detection enabled
• Strings in SectionType::StringData receive a confidence boost for all semantic patterns

Symbol Classification#

Import and export symbols receive special handling through the dedicated SymbolClassifier 67:

pub struct SymbolClassifier {
    known_apis: HashSet<String>, // generic API names
    crypto_apis: HashSet<String>, // cryptographic API names
    network_apis: HashSet<String>, // networking API names
}

The classifier:

1. Tags symbols as Tag::Import or Tag::Export based on their source
2. Applies additional semantic tags based on API name membership (e.g., Tag::Crypto, Tag::Network)
3. Returns both tags for use by the ranking engine

Rust Symbol Demangling#

use rustc_demangle::demangle;

pub fn classify_rust_symbol(mangled: &str) -> Vec<Tag> {
    let mut tags = vec![Tag::Export];
    if let Ok(demangled) = demangle(mangled) {
        let s = demangled.to_string();
        if s.contains("::main") { tags.push(Tag::EntryPoint); }
        if s.contains("panic") { tags.push(Tag::ErrorHandling); }
    }
    tags
}

68

Design Note: The FoundString data model carries an original_text: Option<String> field that preserves the mangled symbol name before demangling. This dual-preservation approach was introduced deliberately: both the human-readable demangled form and the original mangled form have investigative value in forensic analysis 20.

C++ / MSVC Symbol Demangling#

In addition to Rust, the cpp_demangle crate was added to handle C++ symbol mangling (including MSVC name-mangling conventions), with patterns organized into a dedicated patterns submodule within src/classification 64.

Confidence Scoring per Tag#

Each tag assignment is given a per-classification confidence score (0.0–1.0) 69:

impl SemanticClassifier {
    fn calculate_confidence(&self, text: &str, tag: &Tag, context: &StringContext) -> f32 {
        let mut confidence = 0.5; // base confidence
        match tag {
            Tag::Url => {
                if text.starts_with("https://") { confidence += 0.3; }
                if self.has_valid_tld(text) { confidence += 0.2; }
            }
            Tag::FilePath => {
                if context.section_type == SectionType::StringData { confidence += 0.2; }
                if self.has_valid_path_structure(text) { confidence += 0.2; }
            }
            // ... other tag-specific adjustments
        }
        confidence.min(1.0)
    }
}

Multi-Pattern Matching#

A single string can carry multiple tags. When multiple patterns match, additional cross-pattern analysis is performed 70. For example, a URL whose query string contains a Base64 segment receives both Tag::Url and Tag::Base64. The ranking engine rewards multi-tagged strings with a bonus.

Language-Specific Patterns#

When the compiler language is known or can be inferred, classification is further specialised 71:

pub enum LanguageHint { Rust, Go, DotNet, Native }

impl SemanticClassifier {
    fn classify_with_language_hint(&self, text: &str, hint: LanguageHint) -> Vec<Tag> {
        match hint {
            LanguageHint::Rust => self.classify_rust_patterns(text),
            LanguageHint::Go => self.classify_go_patterns(text),
            LanguageHint::DotNet => self.classify_dotnet_patterns(text),
            LanguageHint::Native => self.classify_native_patterns(text),
        }
    }
}

False Positive Reduction#

The classifier uses several techniques to suppress false positives 72:

1. Length thresholds — very short matches are suppressed (Domain < 4 chars, Base64 < 8 chars)
2. Context validation — surrounding data must support the classification
3. Entropy cross-check — high-entropy strings likely represent binary data, not semantic content
4. Whitelist/blacklist — known-good and known-bad patterns

Performance#

• All regex objects are reused via static initialisation — no per-call compilation overhead
• Batch classification is parallelised via rayon::par_iter() 73
• String interning is used for common patterns to reduce memory allocation

Ranking and Scoring Model#

Source: docs/src/ranking.md

The ranking system is the engine that surfaces the most meaningful strings first. It combines four independent scoring components into a single integer score in the range 0–100.

Scoring Formula#

Final Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty

74

The raw result is clamped to [0, 100] 75.

Component 1: Section Weight#

The section a string comes from is the strongest single predictor of its value. Section weights are set by the SectionType classification produced during container parsing 76:

Format-specific bonuses apply on top of the base weight 77:

let format_bonus = match (format, section_name) {
    (BinaryFormat::Elf, ".rodata.str1.1") => 5, // Aligned string pool
    (BinaryFormat::Pe, ".rsrc") => 5, // Rich resource section
    (BinaryFormat::MachO, "__TEXT,__cstring") => 5, // Dedicated string pool
    _ => 0,
};

So a string found in .rodata.str1.1 of an ELF binary gets 40 + 5 = 45 from section weight alone.

Component 2: Encoding Confidence#

Encoding quality contributes a small but meaningful bonus 78:

ASCII / UTF-8:

UTF-16:

The encoding confidence score ties directly into the noise-filtering confidence calculated during extraction (see § String Extraction — Noise Filtering System).

Component 3: Semantic Boost#

This is the highest-variance component — strings carrying semantic meaning receive large bonuses 79:

Multi-tag bonus: strings with more than one semantic tag receive (tag_count - 1) × 3 additional points 80. A string tagged as both url and b64 gets 25 + 10 + 3 = 38 in semantic boost.

Context-aware multiplier: strings in StringData or Resources sections have their semantic boost multiplied by 1.2; strings in symbol contexts receive an additional +5 81.

Component 4: Noise Penalty#

Various quality signals produce negative adjustments 82.

Entropy Penalty#

High Shannon entropy indicates random or encrypted binary data 83:

if entropy > 4.5 { -15 }
else if entropy > 3.8 { -8 }
else { 0 }

Length Penalty#

Very long strings are often structured data, not readable text 84:

Repetition Penalty#

Strings with repetitive patterns are likely padding or binary structures 85:

Common Noise Pattern Penalty#

Known noise patterns are directly penalised 86:

• All-whitespace / null strings → −20

• 80% hex digit characters (hex dump pattern) → −10

• 3 tabs or >5 commas (table-like data) → −8

Complete RankingEngine Implementation#

pub struct RankingEngine { config: RankingConfig }

impl RankingEngine {
    pub fn calculate_score(&self, string: &FoundString, context: &StringContext) -> i32 {
        let section_weight = self.calculate_section_weight(context);
        let encoding_confidence = self.calculate_encoding_confidence(string);
        let semantic_boost = self.calculate_semantic_boost(&string.tags, context);
        let noise_penalty = self.calculate_noise_penalty(string);

        let raw = section_weight + encoding_confidence + semantic_boost + noise_penalty;
        raw.max(0).min(100)
    }
}

87

Score Interpretation Table#

88

Filtering Recommendations by Use Case#

89

Configuration#

All weights and thresholds are configurable at runtime 90:

pub struct RankingConfig {
    pub section_weights: HashMap<SectionType, i32>,
    pub semantic_boosts: HashMap<Tag, i32>,
    pub entropy_threshold: f32, // Default: 4.5
    pub length_penalty_threshold: usize, // Default: 200
    pub repetition_threshold: f32, // Default: 0.5
}

Debug Mode: Score Transparency#

When debug mode is active, calculate_score populates intermediate breakdown fields on each FoundString — section_weight, semantic_boost, and noise_penalty — so analysts can see exactly why a string received a particular score 91. In normal mode, only the aggregate score field is populated.

API Stability Design#

Both RankingConfig and RankingEngine are marked #[non_exhaustive] to allow future field additions without breaking exhaustive pattern matches in downstream code 92. The rank_strings method uses stable sort to ensure deterministic ordering when scores tie 92.

Binary Format Awareness#

Source: docs/src/binary-formats.md

Stringy's "format-first" design principle means every extraction decision is informed by deep knowledge of how each binary format stores and references strings. This section documents the format-specific implementation for ELF, PE, and Mach-O.

Format Detection#

All three formats are detected using a single call to goblin::Object::parse() 93:

pub fn detect_format(data: &[u8]) -> BinaryFormat {
    match Object::parse(data) {
        Ok(Object::Elf(_)) => BinaryFormat::Elf,
        Ok(Object::PE(_)) => BinaryFormat::Pe,
        Ok(Object::Mach(_)) => BinaryFormat::MachO,
        _ => BinaryFormat::Unknown,
    }
}

ELF (Executable and Linkable Format — Linux / Unix)#

Used primarily on Linux and other Unix-like systems 94.

Key Sections#

95

Section Classification#

The ELF parser classifies sections based on the SHF_EXECINSTR flag before checking names 96:

fn classify_section(section: &SectionHeader, name: &str) -> SectionType {
    // Code sections are identified by flags, not name
    if section.sh_flags & SHF_EXECINSTR != 0 {
        return SectionType::Code;
    }
    match name {
        ".rodata" | ".rodata.str1.1" => SectionType::StringData,
        ".data.rel.ro" => SectionType::ReadOnlyData,
        // ...
    }
}

Symbol Extraction — ✅ Implemented#

The ELF parser provides comprehensive symbol extraction 97:

Symbol types supported:

• STT_FUNC — function symbols
• STT_OBJECT — data object symbols
• STT_TLS — thread-local storage variables
• STT_GNU_IFUNC — indirect functions

Import detection — Identifies all undefined symbols (SHN_UNDEF) that need runtime resolution, handling both STB_GLOBAL and STB_WEAK bindings.

Export detection — Extracts globally visible defined symbols, filtering out STV_HIDDEN and STV_INTERNAL symbols to expose only what is externally visible.

Library dependency extraction — Parses DT_NEEDED entries from the dynamic section to enumerate required shared libraries.

Symbol-to-Library Mapping#

One of the harder problems in ELF analysis is attributing an imported symbol to a specific library. Stringy implements best-effort attribution using ELF version tables 98:

Symbol Index → versym[sym_index] → version_index → verneed lookup → library_name

The version tables work as follows:

1. versym table — maps each dynamic symbol to a version index (0=local, 1=global, ≥2=versioned)
2. verneed table — maps version indices to library filenames from DT_NEEDED

Fallback strategies when version information is absent 99:

• For unversioned symbols: attempt to match common symbols (e.g., printf, malloc) to libc
• If only one library is needed: attribute to that library (least accurate)
• Otherwise: return None to avoid false positives

Limitations of ELF's indirect linking model 100:

• Version-based mapping is accurate only when version information is present
• Symbols without version info cannot be definitively mapped without PLT/GOT relocation analysis
• Stripped binaries may lack symbol tables entirely
• Static binaries have no dynamic section and all imports resolve to library: None

Design Note: The current implementation is sufficient for string classification use cases where approximate library attribution is acceptable. Definitive mapping would require complex PLT/GOT relocation analysis, which is out of scope for the MVP.

PE (Portable Executable — Windows)#

Used on Windows for executables (.exe), libraries (.dll), and drivers (.sys) 101.

Key Sections#

102

Import / Export Extraction#

Imports and exports are extracted via goblin's parsed structures 103:

• Imports: iterates pe.imports, creating ImportInfo with name, DLL library, and RVA
  • Example: printf from msvcrt.dll
• Exports: iterates pe.exports, creating ExportInfo with name, address, and ordinal
  • Handles unnamed exports with "ordinal_{i}" naming
  • Note: PE executables typically export nothing; DLLs do

Resource Extraction — ✅ Phase 2 Complete#

The .rsrc section is a uniquely rich source of strings in PE files. Stringy implements structured parsing for three resource types:

VERSIONINFO#

Uses pelite's high-level version_info() API to extract all StringFileInfo key-value pairs, supporting multiple language variants via the translation table 104:

Commonly extracted fields:

• CompanyName, FileDescription, FileVersion, ProductName
• ProductVersion, LegalCopyright, InternalName, OriginalFilename

All strings are UTF-16LE encoded in the resource. Tagged with Tag::Version and Tag::Resource.

STRINGTABLE#

Parses RT_STRING resources (type 6), which store localised UI strings in blocks of 16 105:

Block ID = (StringID >> 4) + 1
String format = u16 length (in UTF-16 code units) + UTF-16LE string data

Multiple language variants are supported. Tagged with Tag::Resource.

MANIFEST#

Extracts RT_MANIFEST resources (type 24) containing application manifests with automatic encoding detection 106:

• UTF-8 with BOM (EF BB BF)
• UTF-16LE with BOM (FF FE)
• UTF-16BE with BOM (FE FF)
• Fallback: byte-pattern analysis

Returns full XML manifest content. Tagged with Tag::Manifest and Tag::Resource.

Section Weight Calculation#

107

Current PE Limitations (Future Work)#

108

Mach-O (Mach Object — macOS / iOS)#

Used on macOS and iOS for executables, frameworks, and libraries 109.

Key Segments and Sections#

110

Load Command Processing#

Mach-O load commands contain valuable string data 111:

Mach-O-Specific Features#

• Two-level namespace: sections are identified by (segment, section) pairs (e.g., ("__TEXT", "__cstring"))
• Fat binaries: multi-architecture fat binaries contain multiple Mach-O slices; configurable via process_fat_binaries = "first" | "all" | <arch>
• String pools: __cstring is a centralised, null-terminated string pool — most readable strings in a macOS binary live here

Section Classification#

impl MachoParser {
    fn classify_section(segment_name: &str, section_name: &str) -> SectionType {
        match (segment_name, section_name) {
            ("__TEXT", "__cstring") => SectionType::StringData,
            ("__DATA_CONST", _) => SectionType::ReadOnlyData,
            ("__DATA", _) => SectionType::WritableData,
            // ...
        }
    }
}

112

Cross-Platform Considerations#

Encoding Differences#

113

String Storage Patterns#

• ELF: strings in .rodata with null terminators, often in an aligned pool
• PE: mix of narrow (ANSI) and wide (UTF-16) strings; resources exclusively UTF-16
• Mach-O: centralised in __cstring, mostly UTF-8 114

Format-Agnostic Weight Normalisation#

fn calculate_section_weight(format: BinaryFormat, section_type: SectionType) -> i32 {
    match (format, section_type) {
        (BinaryFormat::Elf, SectionType::StringData) => 10, // .rodata
        (BinaryFormat::Pe, SectionType::Resources) => 9, // .rsrc
        (BinaryFormat::MachO, SectionType::StringData) => 10, // __cstring
        // ...
    }
}

115

Future Format Support#

Planned extensions 116:

• WebAssembly (WASM) — growing importance in web and edge computing
• Java Class Files — JVM bytecode analysis
• Android APK/DEX — mobile application analysis
• ARM64 enhancements — pointer authentication, tagged pointers
• ELF note sections — build IDs, GNU attributes

Output Formats#

Source: docs/src/output-formats.md

Stringy supports three primary output formats serving distinct workflows. All formats present the same underlying FoundString data with different structure and emphasis.

Format Comparison#

117

Human-Readable Format (Default)#

The default output is an interactive table view optimised for manual analysis 118:

Score Offset Section Encoding Tags String
----- ------ ------- -------- ---- ------
  95 0x1000 .rdata utf-8 url,https https://api.example.com/v1/users
  87 0x2000 .rdata utf-8 guid {12345678-1234-1234-1234-123456789abc}
  82 0x3000 __cstring utf-8 filepath /usr/local/bin/application
  78 0x4000 .rdata utf-8 fmt Error: %s at line %d
  75 0x5000 .rsrc utf-16le version MyApplication v1.2.3

TTY mode features:

• Colour coding: scores ≥ 80 in green, 50–79 in yellow, < 50 in red
• Truncation: long strings trimmed with … indicator
• Alignment: columns properly padded for readability
• Sorting: results sorted by score descending (highest first)

Plain mode (non-TTY / NO_COLOR=1): same columns, no ANSI escape codes.

Usage:

stringy binary # TTY detection is automatic
stringy --format human binary # Explicit

JSON Lines Format (JSONL)#

Machine-readable format with one JSON object per line — compatible with jq, grep, and streaming pipelines 119:

{"text":"https://api.example.com/v1/users","encoding":"utf-8","offset":4096,"rva":4096,"section":".rdata","length":31,"tags":["url"],"score":95,"source":"SectionData"}
{"text":"{12345678-1234-1234-1234-123456789abc}","encoding":"utf-8","offset":8192,"rva":8192,"section":".rdata","length":38,"tags":["guid"],"score":87,"source":"SectionData"}

Schema#

120

Optional debug fields (present when --debug is active): original_text, section_weight, semantic_boost, noise_penalty. Fields are omitted when null (serialisation uses skip_serializing_if = "Option::is_none").

Pipeline Integration Examples#

# Extract only URLs
stringy --json binary | jq 'select(.tags[] == "url") | .text'

# High-confidence strings only
stringy --json binary | jq 'select(.score > 80)'

# Group by section
stringy --json binary | jq -r '.section' | sort | uniq -c

# Find strings in a specific section
stringy --json binary | jq 'select(.section == ".rdata")'

# Export to file
stringy --json binary > strings.jsonl

121

Usage:

stringy --json binary # Short form
stringy --format json binary # Explicit

YARA Format#

Specialised output for creating YARA detection rules 122. Only high-confidence strings (score ≥ 80 by default) are included:

/*
 * Stringy extraction from: binary.exe
 * Generated: 2024-01-15 10:30:00 UTC
 * High-confidence strings (score >= 80)
 */

rule binary_exe_strings {
    meta:
        description = "Strings extracted from binary.exe"
        generated_by = "stringy"

    strings:
        // URLs (score: 95)
        $url_1 = "https://api.example.com/v1/users" ascii wide

        // GUIDs (score: 87)
        $guid_1 = "{12345678-1234-1234-1234-123456789abc}" ascii wide

        // File paths (score: 82)
        $path_1 = "/usr/local/bin/application" ascii

        // Format strings (score: 78)
        $fmt_1 = "Error: %s at line %d" ascii

    condition:
        any of them
}

YARA-specific features:

• Special character escaping: handles backslashes, quotes, and control characters
• Hex encoding: binary strings that cannot be safely represented as literals are converted to hex format
• Encoding modifiers: ascii or ascii wide based on the string's detected encoding (UTF-16 strings receive wide)
• Semantic grouping: strings are grouped by category with inline score comments
• Metadata block: includes extraction timestamp, source filename, and generator attribution
• Score threshold: defaults to ≥ 80; configurable via --min-score

Usage:

stringy --yara binary # YARA format
stringy --format yara binary # Explicit
stringy --yara --min-len 8 binary # Longer strings only

Output Customisation#

All three formats support the same filtering options 123:

# Limit to top N results
stringy --top 50 --format json binary

# Filter by semantic tag
stringy --only url,domain --format yara binary

# Minimum score threshold (post-processing via jq)
stringy --json binary | jq 'select(.score >= 70)'

# Redirect to file
stringy --json binary > strings.jsonl
stringy --yara binary > rules.yar

Planned Future Formats#

Three additional output formats are planned 124:

CSV — simple tabular export compatible with spreadsheet tools:

text,encoding,offset,section,tags,score
"https://api.example.com",utf-8,4096,.rdata,"url",95

XML — structured format with nested tag support:

<strings>
  <string offset="4096" section=".rdata" encoding="utf-8" score="95">
    <text>https://api.example.com</text>
    <tags><tag>url</tag></tags>
  </string>
</strings>

Markdown — human-readable report suitable for documentation:

# String Analysis Report
## High Confidence (Score >= 80)
### URLs
- `https://api.example.com` (score: 95, offset: 0x1000, section: .rdata)

Configuration and Performance#

Sources: docs/src/configuration.md, docs/src/performance.md

Configuration#

Stringy's configuration system spans extraction, classification, ranking, output, and per-format settings. It is designed to be specified via TOML config file, CLI flags, or environment variables, with environment variables taking highest precedence.

Note: Configuration file support (~/.config/stringy/config.toml) is planned but not yet implemented. CLI flags are currently the primary configuration mechanism 125.

Configuration File Structure#

[extraction]
min_ascii_len = 4 # Minimum ASCII string length (CLI: --min-len)
min_utf16_len = 3 # Minimum UTF-16 string length
max_string_len = 1024 # Upper bound (prevents memory issues)
encodings = ["ascii", "utf16le"] # Active encodings
include_debug = false # Include debug section strings
include_symbols = true # Include import/export names
demangle_rust = true # Demangle Rust symbols
demangle_cpp = false # Demangle C++ symbols (future)

[classification]
detect_urls = true
detect_domains = true
detect_ips = true
detect_paths = true
detect_guids = true
detect_emails = true
detect_base64 = true
detect_format_strings = true
min_confidence = 0.7

[ranking]
section_weight_multiplier = 1.0
semantic_boost_multiplier = 1.0
noise_penalty_multiplier = 1.0

[output]
format = "human" # human, json, yara
max_results = 100
show_scores = true
show_offsets = true
color = true

126

Extraction Configuration#

String length limits 127:

• min_ascii_len: lower bound on ASCII string length; increasing this reduces noise significantly
• min_utf16_len: lower bound for UTF-16 strings (default 3 vs ASCII's 4, because UTF-16 detection is already stricter)
• max_string_len: prevents runaway memory from very long strings

Encoding selection 128:

• ascii — 7-bit ASCII
• utf8 — UTF-8 (superset of ASCII)
• utf16le — UTF-16 Little Endian (Windows default)
• utf16be — UTF-16 Big Endian (Java, network)

Section filtering 129:

[extraction]
include_sections = [".rodata", ".rdata", "__cstring"]
exclude_sections = [".debug_info", ".comment"]
include_debug = false
include_resources = true

Classification Configuration#

Per-pattern confidence thresholds allow tuning precision vs recall 130:

[classification]
min_confidence = 0.7 # Overall floor
url_min_confidence = 0.8 # URL-specific threshold
domain_min_confidence = 0.75 # Domain-specific threshold
path_min_confidence = 0.6 # File path threshold

Custom patterns allow adding organisation-specific indicators without code changes 131:

[classification.custom_patterns]
api_key = 'api[_-]?key["\s]*[:=]["\s]*[a-zA-Z0-9]{20,}'
jwt_token = 'eyJ[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+'
crypto_addr = '(bc1|[13])[a-zA-HJ-NP-Z0-9]{25,62}'

Ranking Configuration#

Section weights and semantic boosts are fully overridable 132:

[ranking.section_weights]
string_data = 40
resources = 35
readonly_data = 25
debug = 15
writable_data = 10
code = 5

[ranking.semantic_boosts]
url = 25
domain = 20
guid = 20

[ranking.penalties]
high_entropy_threshold = 4.5
high_entropy_penalty = -15
repetition_threshold = 0.7
repetition_penalty = -12

Format-Specific Configuration#

PE-specific options 133:

[formats.pe]
extract_version_info = true
extract_manifests = true
extract_string_tables = true
prefer_utf16 = true

ELF-specific options 134:

[formats.elf]
include_build_id = true
process_dynamic_strings = true
include_note_sections = false

Mach-O-specific options 135:

[formats.macho]
process_load_commands = true
include_framework_paths = true
process_fat_binaries = "first" # first, all, or specific arch

Environment Variables#

136

Predefined Profiles#

Three profiles cover the most common use cases 137:

Security analysis — focus on high-value IoCs:

stringy --profile security malware.exe
# Equivalent to: min_ascii_len=6, encodings=[ascii,utf8,utf16le],
# only_tags=[url,domain,ipv4,ipv6,filepath,regpath], min_score=70

YARA development — long, high-confidence strings for rules:

stringy --profile yara suspicious.dll
# Equivalent to: format=yara, min_ascii_len=8, exclude_tags=[import,export],
# min_score=80, max_results=50

Development — maximum coverage for debugging:

stringy --profile dev application
# Equivalent to: include_debug=true, include_symbols=true, max_results=500

Performance#

Typical Processing Times#

138

Memory Management#

Stringy uses memory mapping (memmap2) for files above the configured threshold (default 10 MB) 139:

if file_size > MEMORY_MAP_THRESHOLD {
    let mmap = unsafe { Mmap::map(&file)? };
    process_data(&mmap[..]) // OS manages paging; no heap allocation for file data
} else {
    let data = std::fs::read(path)?;
    process_data(&data)
}

Peak memory formula 140:

Peak Memory = Base (~5–10 MB)
            + File Size (if not memory-mapped)
            + String Storage (~2–5× total extracted string length)
            + Classification Data (~1–2 MB for regex + caches)

Memory reduction strategies:

stringy --max-len 200 large_file.exe # Cap string length
stringy --top 100 large_file.exe # Limit result set
stringy --sections .rodata large_file # Process specific sections only

CPU Complexity#

The core pipeline has well-understood complexity 141:

Key CPU optimisation — regex caching 142:

lazy_static! {
    static ref URL_REGEX: Regex = Regex::new(r"https?://[^\s]+").unwrap();
    static ref DOMAIN_REGEX: Regex = Regex::new(r"[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}").unwrap();
}

All classification patterns are compiled once at startup. The Rust regex engine is deterministic and does not have catastrophic backtracking.

Common Bottlenecks#

143

1. Large UTF-16 sections — UTF-16 confidence scoring is more CPU-intensive than ASCII scanning; use --enc ascii when wide strings are not expected
2. Many small strings — per-string classification overhead; increase --min-len to reduce volume
3. Complex custom regex patterns — custom patterns with .* quantifiers can be expensive
4. Large JSON output — serialisation and I/O become the bottleneck for results with thousands of strings; use --top to limit

Storage Performance#

144

Optimisation Recipes#

Fast interactive scan — high-value sections only 145:

stringy --sections .rodata,.rdata --top 20 binary.exe
stringy --enc ascii --min-len 6 binary.exe

Batch processing — maximise throughput 146:

find /binaries -name "*.exe" -exec stringy --json {} \; > all_strings.jsonl

# Parallel with xargs
find /binaries -name "*.so" | xargs -P 4 -I {} stringy --json {} > results.jsonl

Large file handling 147:

stringy --sections .rodata,.rdata,.rsrc --min-len 8 --top 50 huge_file.exe

Focused security scan 148:

stringy \
  --enc ascii,utf8 \
  --min-len 8 \
  --only url,domain,ipv4,filepath \
  --top 20 \
  --sections .rodata,.rdata \
  malware.exe

Built-in Observability#

# Enable per-stage timing
stringy --timing binary.exe

# Unix memory profiling
/usr/bin/time -v stringy large_file.exe

# Cargo benchmarks
cargo bench --bench extraction
cargo bench --bench classification

149

Performance Roadmap#

150

Design Plans and Rationale#

Sources: .kiro/specs/stringy-binary-analyzer/design.md, requirements.md, tasks.md, concept.md, and the PR history 151 64 152 50 20 92 91

This section documents the major architectural decisions made during Stringy's design and implementation, including the why behind each choice.

Design Philosophy: Format-First#

The central thesis of Stringy is articulated in the specification 153:

Stringy is a smarter alternative to the standard strings command that leverages format-specific knowledge to distinguish meaningful strings from random garbage data. The core innovation is understanding enough about each file format to know what constitutes a legitimate string versus noise.

This is expressed as the Format-First Approach principle 154: every extraction decision is informed by file format knowledge. There is no generic "scan all bytes" fallback. The consequence is that code which looks like it could be simplified to a raw byte scan should not be — it should use section context, section type, and encoding metadata.

Symbol Classification Pipeline#

What it does: extends symbol analysis beyond basic function extraction to cover the full range of ELF symbol types and enable symbol-to-library attribution.

Design decisions 151:

1. Comprehensive symbol type coverage — The ELF parser was extended from extracting only STT_FUNC symbols to supporting STT_OBJECT (data objects), STT_TLS (thread-local storage), and STT_GNU_IFUNC (indirect functions). Rationale: a symbol classifier that only knows about functions misses a significant fraction of the binary's import surface, weakening downstream analysis.

2. Visibility filtering — Export filtering respects ELF symbol visibility: STV_HIDDEN and STV_INTERNAL symbols are excluded. Rationale: these are implementation details not intended to be part of the public ABI; surfacing them creates noise and confusion for analysts.

3. Library dependency extraction — extract_needed_libraries() parses DT_NEEDED entries, returning the list of required shared libraries. Rationale: this provides the foundation for symbol-to-library attribution, even when PLT/GOT relocation analysis is not available.

4. Best-effort library attribution via version tables — Symbol-to-library mapping uses the versym/verneed ELF version tables:

   ⧉

   Symbol → versym[sym_index] → version_index → verneed lookup → library_name
   

   Rationale: definitive attribution would require parsing PLT/GOT relocations, which is architecturally complex. Version-table mapping is accurate for the majority of binaries that use symbol versioning, without requiring a full relocation analysis pass.

5. Graceful fallback — When version information is unavailable, the system falls back to heuristic matching (common libc symbols), single-library attribution, or None. Rationale: returning None on ambiguity is safer than a wrong attribution. False positives in library attribution would mislead analysts.

Requirement traceability: Requirements 4.2, 4.3 155, Tasks 3.1, 10.1 156.

MSVC Symbol Demangling#

What it does: extends symbol demangling from Rust-only to C++ (including MSVC mangling), with preservation of the pre-demangling symbol text.

Design decisions 64 20:

1. Modular patterns structure — Classification patterns were reorganised into a patterns module within src/classification, with submodules for IP addresses, network indicators, paths, and data formats. Rationale: as classification rules grew, a flat semantic.rs file became unwieldy. Modular submodules allow each domain expert to own their pattern set.

2. Adding cpp_demangle alongside rustc-demangle — Both crates are now dependencies. Rationale: binary analysis in practice involves binaries compiled in C++, and MSVC-mangled names are unintelligible without demangling.

3. original_text: Option<String> field — The FoundString struct was extended to preserve the original mangled symbol text alongside the demangled form. Rationale: both forms have investigative value. The demangled form is human-readable; the original mangled form is what disassemblers and other tools will display, so analysts need it to cross-reference findings. This is an intentional dual-preservation strategy for forensic-grade analysis 20.

4. Serialisation strategy — Optional fields use #[serde(skip_serializing_if = "Option::is_none", default)]. Rationale: this keeps JSON output compact by default while allowing older schemas to deserialise new fields gracefully, preserving backwards compatibility.

Requirement traceability: Requirement 4.1 157, Task 10 158.

Encoding-Confidence Close-Out#

What it does: fixes a fundamental correctness bug in UTF-16 byte-order confidence scoring, upgrades deduplication to O(1), and improves error types.

The bug 50: The original check_byte_order_consistency() function operated on decoded u16 values. These values are identical regardless of whether the bytes were read as LE or BE — the byte-order interpretation only matters before decoding. The result was that LE always received a consistency score of 1.0 and BE always received 0.0, making it impossible to correctly detect BE strings in Auto mode.

Fixes applied 50:

1. Removed the broken heuristic and redistributed its weight to the remaining, valid confidence components. Rationale: a heuristic that systematically produces wrong results is worse than no heuristic at all.

2. Fixed UTF-16 length semantics — Changed minimum/maximum length enforcement from counting scalar values to counting code units (the byte-pair count). Rationale: the configuration is expressed in characters (code units), not Unicode scalars, so the check must match.

3. O(n²) → O(1) deduplication — The Auto byte-order mode was deduplicating using a linear scan. Replaced with a HashMap<String, ...> keyed by canonical text. Rationale: O(n²) deduplication degrades severely on binaries with many strings; the HashMap is both correct and performant.

4. Replaced Result<_, ()> with StringyError::ParseError — Internal UTF-16 decode functions were using the unit type as an error. Rationale: typed errors carry diagnostic context and propagate cleanly through the error chain. Using () is an anti-pattern that obscures failures.

Lesson for maintainers: this cleanup was driven by real-world testing against diverse binaries. The design documents described the intent correctly; the implementation diverged. When implementing new confidence heuristics, validate them against both LE and BE ground-truth test cases before shipping.

Requirement traceability: Requirement 2.3, 2.4 159, Task 7.1 160.

API Stability: #[non_exhaustive]#

Both RankingConfig and RankingEngine are marked #[non_exhaustive] 92. This prevents downstream code from using struct literal construction or exhaustive pattern matching against these types.

Rationale: the ranking formula and its parameters are expected to evolve as more signal sources are added (e.g., cross-reference analysis, DWARF data). Without #[non_exhaustive], adding a field to RankingConfig would be a breaking change for any downstream code that constructs it with struct literals. With it, new fields can be added in a minor version.

The same principle applies to FoundString — the #[non_exhaustive] attribute there 20 ensures that adding debug fields (section_weight, etc.) does not break code that constructs test fixtures.

Debug Score Transparency#

The ranking engine populates per-component breakdown fields on FoundString only when debug mode is active 91:

debug=off: only `score` is set
debug=on: `score`, `section_weight`, `semantic_boost`, `noise_penalty` all set

Rationale: always computing and storing three extra i32 fields would add memory and serialisation overhead for every string in every run. Since breakdown data is only useful during development and debugging, it is gated behind an explicit flag. This is an example of the lazy evaluation principle applied to diagnostic data.

Extensibility via Feature Gates#

Requirement 9 mandates that new file format support be controlled by feature flags 161:

When feature gates are disabled, the system SHALL not include the corresponding parser code in the binary.

The ContainerParser trait enables this: each format parser is a separate module. A format's parser can be conditionally compiled with #[cfg(feature = "wasm")] etc., keeping the core binary lightweight while allowing opt-in expansion.

Stable Sort for Deterministic Output#

rank_strings uses stable sort 92. When two strings have identical scores, their relative order in the output is the same as their order from the extraction step (which is section order, then byte offset).

Rationale: deterministic output is essential for diffing analysis runs and for snapshot tests. An unstable sort would produce different orderings for tied strings across runs, making regression detection impossible.

Implementation Status Summary#

The tasks tracker 162 documents current progress:

Original Development Roadmap#

The concept.md document defines the incremental delivery plan 163:

Future enhancements from the concept 164:

• XREF hinting — for ELF, check relocations targeting .rodata; strings with inbound relocs rank higher
• Capstone pass — scan .text for immediates pointing into string pools; mark as "referenced"
• UPX detection — detect packers; offer --expect-upx mode
• DWARF skim — function/file names via gimli to augment context
• PDB integration — use pdb crate to enrich PE import names (no symbol server)
• Red team features — --diff, --mask common, --profile malware