Test File Location and Visibility Boundaries#

In Rust projects, test placement determines what code the tests can access. libmagic-rs enforces a strict convention: unit tests live in #[cfg(test)] modules adjacent to source files, while integration tests live in the tests/ directory. This distinction is not cosmetic — it is enforced by Rust's visibility system. The tests/ directory compiles as an external crate and can only access pub items; tests in source files compile as part of the same crate and can access pub(crate) and private items through use super::* 1.

The project uses this boundary deliberately to maintain API encapsulation. Internal implementation details — parser preprocessing stages, anchor-threading for relative offset evaluation, error mapping utilities — are marked pub(crate), making them testable from unit tests but invisible to integration tests and external consumers 2 3. The anti-pattern of widening visibility from pub(crate) to pub just to enable testing is explicitly avoided: it leaks implementation details into the public API and locks those details into the semver contract.

This article documents the visibility boundary rules, the unit/integration test split convention in libmagic-rs, concrete examples of pub(crate) usage, guidance for placing tests based on what they access, doctest constraints, and the conditional module pattern used for build-script testing.

Rust Test Location and Visibility#

The Three-Way Split#

Rust provides three distinct compilation contexts for tests, each with different visibility access:

Unit Tests: Full Crate Access#

Unit tests in libmagic-rs are defined using the #[cfg(test)] pattern. The test code lives either inline at the bottom of a source file or in a dedicated peer file (e.g., src/evaluator/tests.rs) declared with #[cfg(test)] mod tests; 4.

The canonical form:

#[cfg(test)]
mod tests {
    use super::*; // imports all items from parent module, including pub(crate)

    #[test]
    fn test_evaluation_context_offset_management() {
        let mut ctx = EvaluationContext::new(EvaluationConfig::default());
        ctx.set_current_offset(42);
        assert_eq!(ctx.current_offset(), 42);
    }
}

5

Because the tests module is a child module of the source file, use super::* grants access to all items in scope, including pub(crate) functions, structs, and fields invisible to external crates.

Key unit test files in libmagic-rs 1:

• src/evaluator/tests.rs — 77 unit tests for EvaluationContext, offset management, recursion depth, and confidence calculation 6
• src/parser/grammar/tests/mod.rs — 91+ unit tests for magic rule parsing, with a submodule for indirect offset parsing 7
• src/evaluator/operators.rs — 86 unit tests for comparison operators
• src/parser/preprocessing.rs — 32 unit tests for the preprocessing pipeline, exercising pub(crate) functions directly 8
• src/evaluator/offset/mod.rs — unit tests for offset resolution, inline in the source file 9
• src/tests.rs — 26 unit tests for EvaluationConfig methods like EvaluationConfig::performance() and EvaluationConfig::comprehensive() 10

Total: approximately 695 unit tests across 22 source files 11.

Integration Tests: Public API Only#

The tests/ directory contains 13 integration test files with approximately 217 test functions 11. These files compile as separate crates and can only access the public API exported through lib.rs. They cannot call pub(crate) functions. Attempting to do so produces Rust error E0603 (private item access) or E0624 (private method accessed).

Integration test files and their scope:

• tests/relative_offset_evaluation.rs — 12 tests for relative offset resolution behavior with GNU file semantics, driving the evaluator through MagicDatabase 11
• tests/cli_integration.rs — CLI binary tests using subprocess-based assert_cmd 12
• tests/indirect_offset_integration.rs — magic file loading and indirect offset evaluation 13
• tests/property_tests.rs — property-based tests with proptest, verifying the evaluator never panics on arbitrary buffers 14
• tests/integration_tests.rs, tests/evaluator_tests.rs, tests/compatibility_tests.rs, and others — end-to-end workflows through MagicDatabase::load_from_file(), MagicDatabase::evaluate_buffer(), and MagicDatabase::with_builtin_rules()

All integration tests use fully qualified paths such as libmagic_rs::MagicDatabase, libmagic_rs::EvaluationConfig, and libmagic_rs::OffsetSpec.

Doctests: External Crate Semantics#

Rustdoc examples in /// comments compile as standalone external crates. They are subject to the same visibility restrictions as tests/ — pub(crate) items are inaccessible and cause E0603.

Doctests must use external crate paths, not crate:::

/// ```rust
/// use libmagic_rs::EvaluationConfig; // correct
/// // use crate::EvaluationConfig; // wrong — crate:: fails in doctests
/// let config = EvaluationConfig::default();
/// ```

The src/lib.rs doctests follow this pattern consistently, using libmagic_rs:: paths throughout 15.

pub(crate) Usage in libmagic-rs#

The project uses pub(crate) to expose internal APIs for unit testing without polluting the public API surface. Key examples:

Parser Internal Infrastructure#

src/parser/mod.rs declares several crate-internal items 2:

pub(crate) mod codegen; // code generation for build-time processing
pub(crate) mod preprocessing; // line preprocessing pipeline

pub(crate) use hierarchy::build_rule_hierarchy;
pub(crate) use preprocessing::preprocess_lines;

src/parser/preprocessing.rs defines the LineInfo struct and its associated functions, all pub(crate) 16:

pub(crate) struct LineInfo {
    pub(crate) content: String,
    pub(crate) line_number: usize,
    pub(crate) is_comment: bool,
    pub(crate) strength_modifier: Option<StrengthModifier>,
}

pub(crate) fn preprocess_lines(input: &str) -> Result<Vec<LineInfo>, ParseError> { ... }
pub(crate) fn parse_magic_rule_line(line: &LineInfo) -> Result<MagicRule, ParseError> { ... }

src/parser/hierarchy.rs exposes the hierarchy builder 17:

pub(crate) fn build_rule_hierarchy(lines: Vec<LineInfo>) -> Result<Vec<MagicRule>, ParseError> { ... }

These functions form a crate-internal parsing pipeline. Unit tests in src/parser/preprocessing.rs exercise them directly 18. Integration tests access the same behavior only through MagicDatabase::load_from_file().

Evaluator Internal APIs#

src/evaluator/offset/mod.rs contains an internal error mapping function 19:

pub(crate) fn map_offset_error(e: &OffsetError, original_offset: i64) -> LibmagicError { ... }

PR #211 (open) adds crate-internal anchor-threading APIs to EvaluationContext for relative offset evaluation 20:

impl EvaluationContext {
    pub(crate) fn last_match_end(&self) -> usize { ... }
    pub(crate) fn set_last_match_end(&mut self, pos: usize) { ... }
}

These methods track where the previous match ended — the anchor used to resolve OffsetSpec::Relative rules. They are intentionally pub(crate) because the anchor-threading contract is an internal engine detail, not part of the stable API surface. A test that needed to call set_last_match_end() directly triggered the visibility boundary issue that prompted this documentation.

The Anti-Pattern: Widening Visibility for Testing#

Never change pub(crate) or private items to pub solely to enable testing from tests/. This defeats the purpose of the crate boundary and enlarges the public API surface that gets locked in at semver time.

Wrong: Widening to pub#

// BAD: made pub only so tests/relative_offset_evaluation.rs compiles
pub fn set_last_match_end(&mut self, pos: usize) { ... }

This approach:

• Leaks the internal anchor-threading contract to external consumers
• Locks the API signature into the semver contract permanently
• Creates maintenance burden: any refactor of internal state management must now be treated as a breaking API change

Correct: Move the Test to src/#

// GOOD: keep pub(crate), place the test where it has crate access
pub(crate) fn set_last_match_end(&mut self, pos: usize) { ... }

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_anchor_near_saturation() {
        let mut ctx = EvaluationContext::new(EvaluationConfig::default());
        ctx.set_last_match_end(usize::MAX);
        assert_eq!(ctx.last_match_end(), usize::MAX);
    }
}

Case Study: PR #211 Relative Offset Evaluation#

During PR #211, a test needed to inject a near-saturation anchor value (usize::MAX) via EvaluationContext::set_last_match_end() to verify that subsequent OffsetSpec::Relative rules skip gracefully without panicking 20. The initial instinct was to add the test to the existing file at tests/relative_offset_evaluation.rs, but that failed with E0624 because set_last_match_end() is pub(crate). The solution was to add the test to src/evaluator/tests.rs, where it has full crate access.

The same PR identified security vulnerability SEC-001: attacker-controlled Pascal-string length prefixes (e.g., \xff\xff\xff\xff) could poison the anchor to usize::MAX, silently suppressing all subsequent relative-offset rules. The fix clamped Pascal-string consumption against remaining buffer size. Testing this path required pub(crate) access to inject the poisoned anchor state directly — reinforcing that the test belongs in src/, not tests/ 20.

Conditional Module Pattern for Build-Script Testing#

Build scripts (build.rs) cannot import the crate they build, so build-script logic cannot be placed in tests/. libmagic-rs solves this by extracting build logic into src/build_helpers.rs, declared conditionally in src/lib.rs 21:

/// Build-time helpers for compiling magic rules.
/// Only available during tests and documentation builds.
#[cfg(any(test, doc))]
pub mod build_helpers;

The build_helpers.rs module contains 35+ unit tests covering error formatting, serialization (type, operator, and value round-trips), code generation, and end-to-end parsing 1. The #[cfg(any(test, doc))] gate means the module compiles into the crate only during test and doc builds — zero runtime overhead in production.

This pattern generalizes: when you need to expose internal state for testing without permanently widening visibility, conditionally exposing a module under #[cfg(any(test, doc))] is preferable to making individual items pub.

Decision Tree for Test Placement#

1. Is the item under test pub?

   • Can live in either src/ unit test or tests/ integration test.
   • Prefer tests/ for workflow-level tests, src/ for low-level unit tests.

2. Is the item pub(crate) or private?

   • Must live in a #[cfg(test)] mod tests block inside the source file.
   • Placing it in tests/ produces E0603 or E0624.

3. Do you need pub(crate) access from tests/?

   • Use #[cfg(any(test, doc))] pub mod ... to conditionally expose a module.
   • Never permanently widen item visibility just to enable testing.

4. Is it a doctest?

   • Only the public API is accessible.
   • Use the full crate path (libmagic_rs::...), not crate::....

Project Convention Summary#

The project enforces >85% overall coverage, >90% for new code, and 100% coverage for parser/evaluator critical paths and all public APIs 1.

Concrete Examples#

Wrong Location: Compile Error#

// tests/my_integration_test.rs
use libmagic_rs::evaluator::EvaluationContext;
use libmagic_rs::EvaluationConfig;

#[test]
fn test_anchor_injection() {
    let mut ctx = EvaluationContext::new(EvaluationConfig::default());
    ctx.set_last_match_end(usize::MAX); // error[E0624]: method is private
}

Rust compiles tests/my_integration_test.rs as an external crate. set_last_match_end() is pub(crate), so it is invisible here.

Correct Location: Compiles Fine#

// src/evaluator/tests.rs
use super::*;
use crate::EvaluationConfig;

#[test]
fn test_evaluate_rules_anchor_near_saturation_skips_relative_child_gracefully() {
    let mut ctx = EvaluationContext::new(EvaluationConfig::default());
    ctx.set_last_match_end(usize::MAX); // fine — same crate
    // ... assert evaluate_rules skips OffsetSpec::Relative rules gracefully
}

20

Integration Test Pattern#

Integration tests drive the library through its public API, testing complete workflows:

// tests/indirect_offset_integration.rs
use libmagic_rs::MagicDatabase;
use tempfile::TempDir;
use std::fs;
use std::io::Write;

#[test]
fn test_indirect_offset_pe_detection_via_magic_file() {
    let temp_dir = TempDir::new().unwrap();
    let magic_path = temp_dir.path().join("pe.magic");

    let mut f = fs::File::create(&magic_path).unwrap();
    writeln!(f, r#"0 string MZ DOS executable"#).unwrap();
    writeln!(f, r#">(0x3c.l) string PE (PE)"#).unwrap();

    let db = MagicDatabase::load_from_file(&magic_path).unwrap();
    let buf = build_pe_like_buffer();
    let result = db.evaluate_buffer(&buf).unwrap();

    assert!(result.description.contains("DOS executable"));
    assert!(result.description.contains("(PE)"));
}

13

CLI Integration Test Pattern (Library Users and rmagic Users)#

CLI tests (for the rmagic binary) run the binary as a subprocess:

// tests/cli_integration.rs
use assert_cmd::Command;
use predicates::prelude::*;
use tempfile::TempDir;

#[test]
fn test_builtin_format_detection() {
    let temp_dir = TempDir::new().expect("Failed to create temp dir");
    let test_file = create_data_file(&temp_dir, "test.elf", b"\x7fELF\x02\x01\x01\x00");

    Command::new(assert_cmd::cargo::cargo_bin!("rmagic"))
        .args(["--use-builtin", test_file.to_str().expect("Invalid path")])
        .assert()
        .success()
        .stdout(predicate::str::contains("ELF"));
}

22

Unit Test in Preprocessing Module#

Unit tests in src/parser/preprocessing.rs call pub(crate) functions directly:

// src/parser/preprocessing.rs — inside #[cfg(test)] mod tests
use super::*;

#[test]
fn test_preprocess_lines_with_comments() {
    let input = "# Comment\n0 string 0 Test";
    let lines = preprocess_lines(input).unwrap();
    assert_eq!(lines.len(), 2);
    assert!(lines[0].is_comment);
    assert!(!lines[1].is_comment);
}

23

Placing this test in tests/ would fail because preprocess_lines is pub(crate) — it is not exported through the public API.

Relevant Code Files#

Related Topics#

• Testing Strategy and Coverage Requirements — coverage targets (>85% overall, 100% for critical paths), tooling (cargo-nextest, cargo-llvm-cov, proptest), and CI integration
• API Design and Semver — what constitutes a breaking change, visibility as a versioning tool
• Parser Architecture — preprocessing pipeline, hierarchy building, and the role of pub(crate) in separating parser stages
• Evaluator Engine — offset resolution modules, anchor-threading for relative offsets, and the EvaluationContext state machine