Testing Strategy and Infrastructure#

The libmagic-rs project maintains a comprehensive, multi-layered testing strategy with 1,200+ test cases across 22 source files with unit tests and 13 integration test files, achieving 94%+ test coverage. The testing architecture combines unit tests, integration tests, property-based testing with proptest, snapshot testing with insta, CLI testing with assert_cmd, and performance benchmarking with criterion. All testing is coordinated through cargo-nextest version 0.9.132 as the primary test runner, with strict coverage requirements enforced through cargo llvm-cov (≥85% overall, >90% for new code). The CI pipeline enforces the 85% coverage threshold with cargo llvm-cov report --fail-under-lines 85.

The infrastructure represents a durable testing architecture designed specifically for a magic file parser and evaluator, featuring unique patterns like build script testing through extracted modules, subprocess-based CLI testing with assert_cmd, cross-platform CLI snapshot normalization, and property-based fuzzing of rule evaluation. Testing tools are version-pinned through mise.toml to ensure consistency between local development and CI environments.

Test Categories and Organization#

Unit Tests#

Unit tests are located in #[cfg(test)] modules within source files, keeping tests close to the code they validate:

• src/lib.rs - 26 unit tests
• src/main.rs - 4 unit tests
• src/evaluator/mod.rs - 77 unit tests
• src/parser/grammar/ - 91 unit tests distributed across submodules (mod.rs, numbers.rs, value.rs)
• src/evaluator/operators.rs - 86 unit tests covering equality, inequality, comparison, and bitwise operations

Total unit test count: approximately 695 tests across all source files.

Integration Tests Directory#

The tests/ directory contains 13 integration test files with approximately 217 test functions:

1. tests/integration_tests.rs (468 lines) - Core end-to-end tests covering the complete workflow: load database, evaluate files/buffers, verify output formatting and metadata. Tests built-in rules, file loading, and custom magic rules. Includes quad type tests: test_quad_lequad_matches_little_endian_value, test_quad_bequad_matches_big_endian_value, test_quad_signed_negative_one, test_quad_nested_child_rule_with_offset.

2. tests/cli_integration.rs (887 lines) - CLI integration tests using subprocess-based testing with assert_cmd. Provides natural process isolation and works correctly under llvm-cov for coverage reporting. Tests command-line interface functionality including multiple file processing, strict mode (--strict), built-in rules (--use-builtin), JSON output, stdin handling, error handling, timeout validation, shell completions, and custom magic files.

3. tests/cli_integration_tests.rs (1,625 lines) - CLI integration tests using the canonical libmagic test suite. Verifies command-line interface functionality by running against third_party/tests/, covering canonical test suite compatibility and compatibility with original libmagic behavior.

4. tests/property_tests.rs (225 lines) - Property-based tests verifying that evaluator never panics, buffer access is bounds-checked, metadata consistency, and serde roundtrips. Includes 6 custom proptest strategies: arb_offset_spec, arb_type_kind, arb_operator, arb_value, arb_magic_rule, arb_buffer.

5. tests/evaluator_tests.rs (248 lines) - Evaluator integration tests through the public MagicDatabase API. Tests confidence calculation, rule ordering, evaluation behavior, and validates that known file types return non-zero confidence scores.

6. tests/compatibility_tests.rs (360 lines) - Compatibility tests ensuring identical results to the original libmagic. Test files are downloaded from the file/file repository and compared against GNU file command.

7. tests/parser_integration_tests.rs (380 lines) - End-to-end integration tests for magic file parser and database integration. Validates the complete flow from file/directory loading through rule evaluation.

8. tests/directory_loading_tests.rs (431 lines) - Integration tests for directory loading functionality. Validates load_magic_directory() function with various directory structures and batch loading of magic files from directories.

9. tests/mime_tests.rs (230 lines) - MIME type mapping integration tests. Tests MIME type detection through the public API including hardcoded mappings, prefix matching, and case insensitivity.

10. tests/tags_tests.rs (163 lines) - Tag extraction integration tests. Tests keyword extraction, case insensitivity, rule path tags, and custom keyword support.

11. tests/json_integration_test.rs (327 lines) - Integration tests for JSON output functionality. Verifies that the CLI correctly integrates the JSON output formatter and produces expected JSON structure when the --json flag is used.

12. tests/cli_normalization.rs (49 lines) - Tests for CLI output normalization functionality. Ensures cross-platform normalization helpers work correctly across different environments using snapshot testing.

13. tests/relative_offset_evaluation.rs (454 lines) - Integration tests for relative offset resolution behavior with GNU file semantics. 12 test cases covering basic relative offsets (Relative(0)), positive/negative deltas, three-level chained matches, string/pstring consumption accounting (NUL terminator and length prefix), top-level zero anchor, sibling propagation, non-matching intermediate siblings preserving anchor state, context reset between buffers, and graceful handling of out-of-bounds/underflow conditions.

Supporting Test Files#

tests/common/mod.rs (185 lines) - Common test utilities for cross-platform compatibility. Provides helpers for normalizing test outputs to ensure consistent snapshot testing across different operating systems. Contains 7 unit tests for the helper functions themselves.

tests/compatibility/README.md (146 lines) - Detailed documentation for compatibility testing. Explains that compatibility tests use test files from the file/file repository as a git submodule.

Testing Tools and Dependencies#

cargo-nextest#

The project uses cargo-nextest version 0.9.132 as the primary test runner for faster, more reliable test execution. All nextest commands use --no-capture to preserve test output visibility.

Snapshot Testing with insta#

CLI functionality is tested using integration tests with insta snapshots to ensure consistent output across different platforms. CLI insta snapshots must use the normalization helper to ensure consistent results between Windows and Unix systems:

mod common;

#[test]
fn test_cli_help_output() {
    let result = run_cli(&["--help"]);
    let stdout = String::from_utf8(result.stdout).unwrap();

    // REQUIRED: Use normalization for CLI snapshots
    let normalized_stdout = common::normalize_cli_output(&stdout);
    assert_snapshot!("help_output", normalized_stdout);
}

The common::normalize_cli_output() function handles executable name conversion (rmagic.exe → rmagic), Windows path prefix removal (\?\), and error message filtering.

CLI Testing with assert_cmd#

The project uses subprocess-based CLI testing with assert_cmd for natural process isolation. The tests/cli_integration.rs file contains comprehensive CLI tests that run the rmagic binary as a subprocess:

use assert_cmd::Command;
use predicates::prelude::*;

#[test]
fn test_builtin_elf_detection() {
    let temp_dir = TempDir::new().expect("Failed to create temp dir");
    let test_file = create_elf_file(&temp_dir);

    Command::new(assert_cmd::cargo::cargo_bin!("rmagic"))
        .args(["--use-builtin", test_file.to_str().expect("Invalid path")])
        .assert()
        .success()
        .stdout(predicate::str::contains("ELF"));
}

This approach provides reliable CLI testing that works correctly under llvm-cov for coverage reporting without platform-specific fd manipulation.

Property-Based Testing with proptest#

The project uses proptest version 1.10.0 for property-based testing to discover edge cases through fuzzing. The tests/property_tests.rs file defines 6 custom strategies:

1. arb_offset_spec() - Generates valid OffsetSpec values (Absolute, Relative, FromEnd)
2. arb_type_kind() - Generates valid TypeKind values (Byte, Short, Long, Quad, String) with different endianness and signedness
3. arb_operator() - Generates valid Operator values including comparison operators
4. arb_value() - Generates valid Value variants (Uint, Int, Bytes, String)
5. arb_magic_rule() - Combines strategies to generate complete MagicRule instances
6. arb_buffer() - Generates arbitrary binary buffers (0-1024 bytes)

Four main property tests verify: evaluation never panics on any buffer, configuration validation consistency, evaluation result metadata validity, and rule serialization roundtrips.

Criterion Benchmarks#

Performance benchmarks are implemented using Criterion in the benches/ directory. Benchmarks run automatically on a weekly schedule (Sundays at 3 AM UTC), on PRs with performance-critical code changes, and via manual workflow_dispatch.

evaluation_bench.rs#

Measures rule evaluation performance for file type detection:

• file_type_detection - Tests detection performance for different file formats: detect_elf, detect_zip, detect_pdf, detect_unknown (fallback behavior)
• buffer_sizes - Tests evaluation performance across different buffer sizes (64 bytes, 256 bytes, 1KB, 4KB, 16KB, 64KB)
• evaluation_configs - Compares performance of different configuration presets: config_default, config_performance, config_comprehensive

io_bench.rs#

Measures file I/O operations and memory-mapped file access:

• file_buffer_creation - FileBuffer creation performance across file sizes (64 bytes to 1MB) with mmap identifiers
• buffer_access - Memory access patterns on FileBuffer: sequential_read_1mb, random_access_pattern (simulates rule evaluation), slice_extraction (common in rule evaluation)
• small_files - Performance for small file handling (8 to 512 bytes) with create_and_access benchmarks including magic number checks

parser_bench.rs#

Measures magic file parsing performance:

• load_builtin_rules - Loading built-in magic rules into MagicDatabase
• magic_file_loading - Loading magic files from disk: load_builtin_magic_file
• rule_parsing - Individual rule pattern parsing: parse_simple_string_rule, parse_numeric_rule (with endianness), parse_nested_rules (complex multi-level ELF detection)

Development Dependencies and Tool Management#

mise for Tool Version Management#

The project uses mise.toml to manage 30+ development tools with pinned versions, including:

• Rust 1.91.0 (with llvm-tools, cargo, rustfmt, clippy)
• cargo-nextest 0.9.132
• cargo-llvm-cov 0.8.5
• cargo-audit, cargo-deny, cargo-dist, and others

All tool commands should be run via mise exec -- to use the project's pinned toolchain.

Dev Dependencies#

Core testing dependencies in Cargo.toml:

• assert_cmd 2.2.1 - Subprocess-based CLI testing with process isolation. In 2.2.1, cargo_bin panics on invalid paths instead of returning a bad path.
• criterion 0.8.2 - Benchmarking framework for performance measurement
• insta 1.47.2 - Snapshot testing library with JSON feature enabled
• nix 0.31.2 - Unix system API bindings with fs feature for filesystem testing
• predicates 3.1.4 - Assertion predicates for command output validation
• proptest 1.10.0 - Property-based testing framework
• regex 1.12.3 - Regular expression support for test assertions
• tempfile 3.26.0 - Temporary file and directory creation for test fixtures

Test Execution Commands#

Basic Test Running#

Test commands from justfile:

# Run all tests with nextest (default via justfile)
just test # cargo nextest run --workspace --no-capture

# Run specific test module
cargo test parser::grammar::tests

# Run specific test
cargo test test_parse_number_positive

# Run tests with output visible
cargo test -- --nocapture

# Run ignored tests
cargo test -- --ignored

Enhanced Test Running#

# Use nextest for faster execution with better output
cargo nextest run

# Run tests with coverage
cargo llvm-cov --html

# Run tests in release mode
cargo test --release

# Test documentation examples
cargo test --doc

Continuous Testing#

# Auto-run tests on file changes
cargo watch -x test

# Auto-run specific tests
cargo watch -x "test parser"

# Run checks and tests together
cargo watch -x check -x test

CI Parity Testing#

The just ci-check command provides full local CI parity, running the complete CI validation suite:

just ci-check

This executes: pre-commit hooks, format checking, linting, test suite, release build, security audit, code coverage validation (≥85%), and distribution plan verification.

Coverage Requirements and Measurement#

Coverage Targets#

The project enforces strict coverage requirements:

• Overall coverage target: >85% for the project
• New code requirement: >90% coverage for new features
• Critical paths: 100% coverage required for parser and evaluator modules
• Public APIs: 100% coverage required for all public functions

Current achievement: 94%+ line coverage across the codebase.

cargo llvm-cov Configuration#

The project uses cargo-llvm-cov version 0.8.5 for coverage measurement, installed via mise.toml. Coverage commands from justfile:

# Generate lcov.info report
just coverage

# Verify coverage meets 85% threshold (fails if below)
just coverage-check

# Generate HTML coverage report for local viewing
just coverage-report

# Show coverage summary by file
just coverage-summary

All coverage commands use RUSTFLAGS="--cfg coverage" and clean the target/llvm-cov-target directory before running.

Codecov Integration#

The codecov.yml configuration defines project-level targets of 80% coverage with 2% variance allowed, and patch-level targets of 85% coverage with 5% variance allowed. CI workflow coverage job generates coverage with --test-threads=1 to prevent race conditions, combines reports, and uploads to Codecov and Qlty services.

Build Script Testing Pattern#

The project implements a unique pattern for testing build script functionality, addressing the challenge that build scripts cannot import the crate being built. Build logic is extracted into src/build_helpers.rs as a library module that uses #[cfg(any(test, doc))] to only compile during tests and documentation builds:

#[cfg(any(test, doc))]
pub mod build_helpers;

The build_helpers.rs module contains 35+ unit tests (lines 346-709) covering error formatting, serialization (serialize_offset_spec, serialize_type_kind, serialize_operator, serialize_value), code generation, and end-to-end parsing. Tests include quad type serialization (test_serialize_type_kind_quad). This pattern provides comprehensive testing of build-time code generation with zero runtime cost.

Test Quality Gates#

All code must pass these quality gates:

1. Zero Warnings: cargo clippy -- -D warnings must pass
2. All Tests Pass: Complete test suite must pass
3. Code Coverage: Target >85% coverage for new code
4. Documentation: All public APIs must be documented
5. Memory Safety: No unsafe code except in vetted dependencies

Overall Testing Architecture#

The comprehensive testing infrastructure leverages cargo-nextest for faster test execution, proptest for property-based testing, and cargo llvm-cov for coverage measurement. Tests are organized with unit tests embedded in source files via #[cfg(test)] modules, integration tests in the tests/ directory, benchmarks in the benches/ directory, and all components coordinated through justfile recipes and CI workflows. All testing commands are executed through mise exec to ensure consistent toolchain versions, and the just ci-check command provides local CI parity for developers.

Relevant Code Files#

Related Topics#

• Magic Rule Parser Architecture
• Evaluator Engine
• CLI Binary (rmagic)
• Build System and Code Generation
• CI/CD Pipeline and Quality Assurance