Testing Strategy and Coverage Requirements#

Overview#

The libmagic-rs project implements a comprehensive, multi-layered testing strategy designed to ensure reliability, maintainability, and cross-platform compatibility for its pure-Rust replacement of libmagic. The testing approach combines traditional unit and integration testing with modern property-based testing techniques, enforcing strict coverage requirements of >85% line coverage.

The project maintains over 900 test cases across 22 source files with unit tests and 12 integration test files, achieving 94%+ test coverage. The testing infrastructure leverages cargo-nextest for faster test execution, proptest for property-based testing, and cargo llvm-cov for coverage measurement.

A unique aspect of the testing strategy is the build script testing pattern, which extracts build logic into testable modules using #[cfg(any(test, doc))] to enable comprehensive testing of build-time code generation. All testing commands are executed through mise exec to ensure consistent toolchain versions, and the just ci-check command provides local CI parity for developers.

Test Organization Structure#

Unit Tests in Source Files#

Unit tests are embedded directly within source files using Rust's #[cfg(test)] module pattern. This keeps tests close to the code they validate and ensures maintainability. The project includes unit tests in 22 source files across all major modules:

Core library tests:

• src/lib.rs - 26 unit tests
• src/main.rs - Core CLI logic (tested via integration tests)
• src/error.rs - 24 unit tests
• src/build_helpers.rs - 35 unit tests

Evaluator module tests:

• src/evaluator/mod.rs - 77 unit tests
• src/evaluator/types.rs - 77 unit tests covering byte, short, long, and quad types with endianness and signedness variations
• src/evaluator/operators.rs - 86 unit tests covering equality, inequality, comparison operators (<, >, <=, >=), and bitwise operations
• src/evaluator/strength.rs - 35 unit tests

Parser module tests:

• src/parser/grammar.rs - 91 unit tests
• src/parser/ast.rs - 36 unit tests
• src/parser/preprocessing.rs - 32 unit tests

The unit test count totals approximately 691 tests across all source files (following CLI test extraction to integration tests).

Integration Tests Directory#

The tests/ directory contains 12 integration test files testing complete workflows:

1. tests/integration_tests.rs (378 lines) - Core end-to-end tests covering built-in rules, file loading, and custom magic rules, including quad type tests (test_quad_lequad_matches_little_endian_value, test_quad_bequad_matches_big_endian_value, test_quad_signed_negative_one, test_quad_nested_child_rule_with_offset)
2. tests/cli_integration.rs (887 lines) - CLI integration tests using assert_cmd for subprocess-based testing. This approach provides natural process isolation and eliminates fd manipulation. Tests cover builtin detection (ELF, PNG, JPEG, PDF, ZIP, GIF), stdin handling with truncation warnings, multiple file processing, error handling, timeout validation, output formats (text/JSON), shell completion, custom magic files, and edge cases like Unicode filenames and empty files.
3. tests/property_tests.rs (221 lines) - Property-based testing with proptest
4. tests/evaluator_tests.rs - Evaluator integration tests for confidence calculation and configuration
5. tests/compatibility_tests.rs - Compatibility with original libmagic implementation
6. tests/parser_integration_tests.rs - Parser integration workflows
7. tests/directory_loading_tests.rs - Directory loading functionality
8. tests/mime_tests.rs - MIME type detection
9. tests/tags_tests.rs - Tag extraction
10. tests/json_integration_test.rs - JSON output format validation
11. tests/cli_normalization.rs - Cross-platform output normalization

Test Helper Utilities#

The tests/common/mod.rs module (185 lines) provides shared test utilities including:

• Cross-platform output normalization functions
• Path handling for Windows/Unix compatibility
• Snapshot test helpers
• 7 unit tests for the helper functions themselves

Coverage Requirements and Tooling#

Coverage Targets#

The project enforces strict coverage requirements:

• Overall coverage target: >85% for the project
• New code requirement: >90% coverage for new features
• Critical paths: 100% coverage required for parser and evaluator modules
• Public APIs: 100% coverage required for all public functions

Current achievement: 94%+ line coverage across the codebase.

cargo llvm-cov Configuration#

The project uses cargo-llvm-cov version 0.6.24 for coverage measurement, installed via mise.toml.

Coverage commands from justfile:

# Generate lcov.info report
just coverage

# Verify coverage meets 85% threshold (fails if below)
just coverage-check

# Generate HTML coverage report for local viewing
just coverage-report

# Show coverage summary by file
just coverage-summary

All coverage commands use RUSTFLAGS="--cfg coverage" and clean the target/llvm-cov-target directory before running to ensure clean builds.

Codecov Integration#

The codecov.yml configuration defines:

Project-level targets:

• Target: 80% coverage
• Threshold: 2% variance allowed

Patch-level targets (new code):

• Target: 85% coverage
• Threshold: 5% variance allowed

Range: 70-95% for coverage coloring

Ignored paths: benches/, tests/, examples/, target/

CI Coverage Workflow#

The CI workflow coverage job:

1. Generates coverage with standard cargo test
2. Combines reports and outputs lcov.info
3. Uploads to Codecov and Qlty services

Test Execution with cargo-nextest#

The project uses cargo-nextest version 0.9.123-b.4 as the primary test runner for faster, more reliable test execution.

Nextest Commands#

Test commands from justfile:

# Standard test execution with output enabled
just test # cargo nextest run --workspace --no-capture

# CI-specific test execution
just test-ci # cargo nextest run --workspace --no-capture

# Run all tests including ignored/slow tests
just test-all # cargo nextest run --workspace --no-capture -- --ignored

All nextest commands use --no-capture to preserve test output visibility.

Nextest Configuration#

The project uses nextest's default configuration without a custom configuration file. No .config/nextest.toml or .nextest.toml exists - the defaults are sufficient for the project's needs.

Nextest is used consistently across all platforms:

• Linux CI tests
• macOS and Windows matrix tests

Coverage exception: The coverage job uses standard cargo test instead of nextest because cargo llvm-cov requires different test execution handling.

Property-Based Testing with proptest#

The project uses proptest version 1.11.0 for property-based testing to discover edge cases through fuzzing.

Custom Proptest Strategies#

The tests/property_tests.rs file defines 6 custom strategies:

1. arb_offset_spec() - Generates valid OffsetSpec values (Absolute, Relative, FromEnd)
2. arb_type_kind() - Generates valid TypeKind values with different endianness and signedness, including TypeKind::Byte { signed }, TypeKind::Short { endian, signed }, TypeKind::Long { endian, signed }, and TypeKind::Quad { endian, signed }
3. arb_operator() - Generates valid Operator values (including comparison operators LessThan, GreaterThan, LessEqual, GreaterEqual)
4. arb_value() - Generates valid Value variants (Uint, Int, Bytes, String)
5. arb_magic_rule() - Combines all strategies to generate complete MagicRule instances
6. arb_buffer() - Generates arbitrary binary buffers (0-1024 bytes)

Property Tests#

The file defines 4 main property tests:

1. prop_evaluation_never_panics - Verifies the evaluator never panics on any buffer, and confidence values are in valid range [0.0, 1.0]

2. prop_config_validation_consistent - Validates that EvaluationConfig accepts reasonable parameter ranges

3. prop_metadata_valid - Ensures evaluation result metadata is consistent with input (file size, evaluation time, rules evaluated)

4. prop_rule_serde_roundtrip - Verifies that arbitrary rules serialize and deserialize consistently

Running Property Tests#

Property tests can be run with configurable test case counts using the PROPTEST_CASES environment variable:

# Run with more test cases for thorough testing
PROPTEST_CASES=10000 cargo test proptest

The CI workflow runs with 1000 cases.

Build Script Testing Pattern#

The project implements a unique pattern for testing build script functionality, addressing the challenge that build scripts (build.rs) cannot import the crate being built.

Pattern Architecture#

1. Build logic extraction - Build logic is extracted into src/build_helpers.rs as a library module

2. Conditional compilation - The module uses #[cfg(any(test, doc))] to only compile during tests and documentation builds:

#[cfg(any(test, doc))]
pub mod build_helpers;

3. Minimal build.rs - The build.rs file contains the actual build logic (379 lines) but parallels the testable implementation

Build Helpers Tests#

The build_helpers.rs module contains 35+ unit tests (lines 346-709) covering:

• Error formatting tests - Parse error message formatting for various error types
• Serialization tests - Tests for serialize_offset_spec(), serialize_type_kind() (including TypeKind::Byte { signed }, TypeKind::Short, TypeKind::Long, and TypeKind::Quad with various endianness options), serialize_operator() (including comparison operators), serialize_value()
• Code generation tests - Empty rules, single rules, nested children serialization
• End-to-end parsing tests - Valid magic file parsing, invalid syntax handling, malformed values

Benefits#

• Comprehensive testing - All build logic can be unit tested, including error paths
• Zero runtime cost - #[cfg(any(test, doc))] ensures the module is excluded from normal builds
• Better error messages - Error formatting can be tested for clear build-time diagnostics
• Code reuse - Same logic serves both build script and test suite

Development Workflow and Tooling#

mise for Tool Management#

The project uses mise.toml to manage 30+ development tools with pinned versions:

Core tools:

• Rust 1.91.0 (with llvm-tools, cargo, rustfmt, clippy)
• Just (command runner)
• Python 3.13.11

Cargo tools:

• cargo-nextest 0.9.123-b.4
• cargo-llvm-cov 0.6.24
• cargo-audit 0.22.0
• cargo-deny, cargo-dist, mdbook, and others

Setup command:

just setup # Installs all tools via mise install

mise exec Pattern#

All tool commands should be run via mise exec -- to use the project's pinned toolchain. The justfile implements this with a mise_exec variable:

mise_exec := "mise exec --"

All commands are prefixed with this variable:

• {{ mise_exec }} cargo fmt --all
• {{ mise_exec }} cargo nextest run --workspace --no-capture
• {{ mise_exec }} cargo clippy --workspace --all-targets --all-features -- -D warnings

just ci-check Command#

The just ci-check command provides full local CI parity, running the complete CI validation suite:

just ci-check

This executes:

1. pre-commit-run - Pre-commit hooks
2. fmt-check - Format checking
3. lint-rust - Rust linting with clippy (all features)
4. lint-rust-min - Rust linting with no default features
5. test-ci - Test suite via cargo nextest
6. build-release - Release build
7. audit - Security audit
8. coverage-check - Code coverage validation (≥85%)
9. dist-plan - Distribution plan verification

Compatibility variant:

just ci-check-compatibility

Also includes compatibility tests against original libmagic.

CI/CD Integration#

GitHub Actions Workflows#

The main CI workflow includes:

Quality job:

• Uses jdx/mise-action@v3.6.3 to install tools
• Runs just lint-rust for formatting and linting

Test job:

• Runs cargo nextest run --all-features
• Builds release: cargo build --release --all-features

Cross-platform test job:

• Matrix testing across Ubuntu (22.04 & latest), macOS, and Windows
• Runs nextest and release builds on each platform

Coverage job:

• Generates coverage with --test-threads=1
• Uploads to Codecov and Qlty services

Additional Workflows#

The project includes additional workflows:

• audit.yml - Security auditing
• benchmarks.yml - Performance benchmarks
• compatibility.yml - Compatibility testing
• docs.yml - Documentation building
• release.yml - Release automation
• security.yml - Security scanning
• scorecard.yml - OpenSSF Scorecard

All workflows use the jdx/mise-action to ensure consistent tool versions matching mise.toml.

Test Patterns and Best Practices#

Integration Test Pattern#

From tests/integration_tests.rs:

#[test]
fn test_builtin_rules_detect_elf() {
    let db = MagicDatabase::with_builtin_rules().unwrap();
    let result = db.evaluate_buffer(b"\x7fELF\x02\x01\x01\x00").unwrap();
    assert!(result.description.contains("ELF"));
    assert!(result.confidence > 0.0);
}

Unit Test Pattern#

From src/evaluator/mod.rs:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_evaluate_byte_equal() {
        let rule = MagicRule {
            offset: OffsetSpec::Absolute(0),
            typ: TypeKind::Byte { signed: false },
            op: Operator::Equal,
            value: Value::Uint(0x7f),
            message: "test".to_string(),
            children: vec![],
            level: 0,
            strength_modifier: None,
        };
        let buffer = vec![0x7f, 0x45, 0x4c, 0x46];
        let config = EvaluationConfig::default();
        let mut context = EvaluationContext::new(config);

        let result = evaluate_rule(&rule, &buffer, &mut context);
        assert!(result.is_some());
    }
}

Snapshot Test Pattern#

From tests/cli_normalization.rs:

#[test]
fn normalizes_executable_suffix_in_snapshots() {
    let input = "Usage: rmagic.exe [OPTIONS] <FILE>";
    let normalized = common::normalize_cli_output(input);
    assert_snapshot!("normalize_exe_suffix", normalized);
}

Cross-Platform Test Utilities#

The tests/common/mod.rs provides normalization helpers:

/// Normalize CLI output for cross-platform snapshot consistency
pub fn normalize_cli_output(input: &str) -> String {
    input
        .replace("rmagic.exe", "rmagic")
        .replace("\\\\?\\", "")
        .lines()
        .filter(|line| !line.contains("error: process didn't exit successfully:"))
        .collect::<Vec<_>>()
        .join("\n")
        .trim()
        .to_string()
}

Related Documentation#

• Testing Guidelines - Comprehensive testing strategy documentation
• Development Guide - Development environment setup
• AGENTS.md - Testing patterns and best practices for AI agents

Relevant Code Files#