Enrichment Pipeline#

The Enrichment Pipeline in opnDossier is a data transformation subsystem that flows from parsed XML configuration (schema.OpnSenseDocument) through converter operations to produce a platform-agnostic CommonDevice model, with final preparation for export via the prepareForExport() function. This pipeline serves as the single gate for all structured format exports (JSON, YAML), computing aggregate statistics, performing security analysis, and applying selective redaction of sensitive fields before serialization.

The enrichment pipeline is architecturally distinct from the primary audit processing pipeline. While the processor package focuses on interactive analysis and markdown report generation, the converter enrichment system operates during export to populate computed fields on the CommonDevice structure. These computed fields—including Statistics, Analysis, SecurityAssessment, and PerformanceMetrics—use pointer types to distinguish them from raw parsed configuration data. The ComplianceResults field, while also a pointer type, is populated externally by the audit handler before enrichment and serves as a pass-through field carrying audit results to the export pipeline.

The pipeline enforces immutability guarantees by creating shallow copies with selective deep-copying of nested structures before applying redaction. This design prevents accidental mutation of the original device configuration, allowing multiple exports with different redaction settings from the same source data.

Architecture and Data Flow#

Pipeline Stages#

The enrichment pipeline follows a multi-stage transformation sequence that begins with raw configuration data and culminates in structured, enriched exports:

1. XML Parsing: Raw OPNsense XML configuration is parsed by internal/cfgparser into schema.OpnSenseDocument (an XML-specific DTO)
2. Device Conversion: The schema is converted to the platform-agnostic CommonDevice model via pkg/parser/opnsense/converter.go. As part of this step, convertKeaDHCPScopes() reads KeaDhcp4.Subnets and KeaDhcp4.Reservations from the XML schema, converts them to DHCPScope entries with Source: "kea", and appends them to CommonDevice.DHCP alongside any ISC scopes.
3. Audit Data Population (when running in audit mode): cmd/audit_handler.go calls mapAuditReportToComplianceResults() to convert audit.Report into device.ComplianceResults, populating compliance findings before enrichment
4. Export Preparation: The prepareForExport() function enriches the device with computed statistics and analysis, optionally applying redaction. This function treats ComplianceResults as a pass-through field—it does not modify or populate it
5. Format Export: Enriched data is serialized to JSON, YAML, or other structured formats

This staged approach separates concerns between parsing (extracting data from XML), normalization (converting to a common model), audit data mapping (converting audit results to the common model), enrichment (computing derived insights), and serialization (formatting for output). The audit data flow ensures compliance results are available to all export formats through the standard enrichment pipeline without requiring the enrichment layer to understand audit-specific types.

CommonDevice Structure#

The CommonDevice struct serves as the central data model for the enrichment pipeline. It organizes data into three categories of fields, each serving a distinct architectural purpose:

Core Configuration Fields (concrete types, populated during parsing):

• System configuration (hostname, DNS, web GUI settings)
• Network infrastructure (Interfaces, VLANs, Bridges, LAGGs, VirtualIPs)
• Firewall rules and NAT configuration
• VPN subsystems (OpenVPN, WireGuard, IPsec)
• Routing (gateways, gateway groups, static routes)
• Services (DHCP []DHCPScope — unified slice covering both ISC and Kea scopes, DNS, NTP, SNMP, load balancer)
• Security (certificates, CAs, IDS/IPS)
• User management (users, groups, authentication)

Optional Features (pointer types for features that may not be configured):

• IDS/IPS (Suricata)
• Monit (process monitoring)
• Netflow/IPFIX
• Traffic shaper (QoS)
• Captive portal
• Cron jobs
• Kea DHCP general settings (KeaDHCP *KeaDHCPConfig — holds server-level settings only; subnet and reservation data flows through DHCP []DHCPScope)

Enrichment Fields (pointer types, populated by prepareForExport):

• Statistics - Aggregate counts and metrics
• Analysis - Configuration analysis findings
• SecurityAssessment - Security posture and vulnerabilities
• PerformanceMetrics - Complexity and performance metrics

Audit Data Fields (pointer types, populated before enrichment by audit handler):

• ComplianceResults - Type: *ComplianceResults. Compliance audit results from plugin-based checks. Populated by mapAuditReportToComplianceResults() in cmd/audit_handler.go before the device enters the enrichment pipeline. The enrichment system treats this field as a pass-through—it does not modify or compute it.

Multi-Format Export Optimization#

The enrichment pipeline provides explicit memoization support for callers preparing the same device for multiple format exports (e.g., JSON + YAML + Markdown) through the EnrichForExport function.

Performance Problem#

analysis.ComputeStatistics and analysis.ComputeAnalysis are O(n) over interfaces, rules, and services. Before PR #608, multi-format exports recomputed these functions once per format because prepareForExport checked for nil enrichment fields on its shallow copy rather than the input device. Exporting the same *CommonDevice to three formats (JSON, YAML, Markdown) paid the O(n) analysis cost three times.

Solution: EnrichForExport#

The EnrichForExport(*CommonDevice) function populates Statistics, Analysis, SecurityAssessment, and PerformanceMetrics directly on the input device when nil. Callers preparing a device for multiple format exports invoke it once before the format loop:

device := parseConfiguration()
EnrichForExport(device) // Heavy analysis runs once

for _, format := range []Format{JSON, YAML, Markdown} {
    output := prepareForExport(device, redact) // Reuses cached fields
    export(output, format)
}

Subsequent prepareForExport calls observe the populated enrichment fields via the shallow copy and skip recomputation. Benchmark measurements on a ~119KB configuration showed ~65% latency and ~65% allocation reduction in the per-format preparation step.

SECURITY Warning#

EnrichForExport does NOT redact sensitive fields. The resulting *CommonDevice carries plaintext secrets — most notably the SNMP community string in Statistics.ServiceDetails — because analysis.ComputeStatistics observes unredacted input by design (presence checks must see real values).

Callers MUST NOT marshal or log the device directly after EnrichForExport. Always pass the device through prepareForExport (or a Generator that calls prepareForExport) so the redaction branch can produce a clone with sensitive fields stripped.

CACHE INVALIDATION Caveat#

EnrichForExport memoizes Statistics and Analysis as a snapshot of the device at call time. If the caller mutates a field that feeds those computations (e.g., device.SNMP.ROCommunity, FirewallRules, Interfaces) after calling EnrichForExport, the cached values go stale and subsequent exports will reflect the pre-mutation state. Re-call EnrichForExport (after first clearing the affected enrichment field) when the underlying configuration changes between exports.

EnrichForExport is not safe for concurrent use on the same *CommonDevice. Callers preparing one device for parallel format exports must call EnrichForExport once before fanning out.

The prepareForExport Function#

Function Signature and Purpose#

The prepareForExport function is the single entry point for enrichment operations before JSON/YAML serialization:

func prepareForExport(data *common.CommonDevice, redact bool) *common.CommonDevice

Parameters:

• data - The input CommonDevice configuration (remains unmodified)
• redact - Boolean flag controlling sensitive field redaction

Returns:

• Pointer to a shallow copy of the input device with enrichment fields populated

Core Responsibilities#

The function performs four critical operations that ensure consistent, secure exports:

1. Enrichment Field Population: Computes and assigns Statistics, Analysis, SecurityAssessment, and PerformanceMetrics if not already present. Note that ComplianceResults is NOT populated by this function—it is treated as a pass-through field that is populated externally by the audit handler before enrichment
2. Redaction Application: Replaces sensitive data with [REDACTED] when redact=true
3. Immutability Guarantee: Creates copies to prevent mutation of the caller's original data
4. Format Consistency: Ensures all JSON/YAML exporters operate on identically enriched data

Copy Strategy#

The function employs a hybrid shallow/deep copy approach that balances performance with safety:

cp := *data // Shallow copy of top-level struct

Before modifying nested structures, it selectively deep-copies affected slices using Go's slices.Clone():

if len(cp.Certificates) > 0 {
    cp.Certificates = slices.Clone(cp.Certificates)
    for i := range cp.Certificates {
        if cp.Certificates[i].PrivateKey != "" {
            cp.Certificates[i].PrivateKey = redactedValue
        }
    }
}

This strategy avoids the performance penalty of deep-copying the entire structure while ensuring mutations don't affect the original data.

Critical Design Decision#

The prepareForExport function now calls the internal enrich() helper on its shallow copy to populate enrichment fields. The enrich() helper is also the core of the public EnrichForExport() function, enabling the memoization pattern described in "Multi-Format Export Optimization."

The function intentionally receives unredacted data for statistics and analysis computation. This ensures presence checks (e.g., "is SNMP configured?") see real values before redaction occurs:

cp := *data // Shallow copy
enrich(&cp) // Populates Statistics/Analysis on the copy

if redact {
    redactSensitiveFields(&cp)
    cp.Statistics = redactStatisticsServiceDetails(cp.Statistics)
}

This two-phase approach—compute from unredacted data via enrich(), then redact the copy—allows statistics to accurately reflect what services are configured while still protecting sensitive details in the final export. The extraction of enrichment logic into enrich() enables both the in-line preparation path (prepareForExport) and the explicit memoization path (EnrichForExport).

Redaction System#

Redaction Pattern#

The redaction system uses a constant marker to replace sensitive values:

const redactedValue = "[REDACTED]"

This consistent pattern makes it easy to identify redacted fields in exported configurations and provides a clear signal that sensitive data has been intentionally removed rather than being empty.

Sensitive Fields Redacted#

The redactSensitiveFields function replaces seven categories of sensitive configuration data:

1. High Availability Password - CARP/pfsync synchronization credentials
2. Certificate Private Keys - TLS/SSL certificate private keys
3. CA Private Keys - Certificate Authority private keys for locally-created CAs
4. API Key Secrets - User API authentication secrets
5. SNMP Community String - SNMP read-only community authentication
6. WireGuard Pre-Shared Keys - VPN tunnel pre-shared keys
7. DHCPv6 Authentication Secrets - DHCP authentication configuration

Never-Mapped Secrets#

As a defense-in-depth measure, additional secrets are vetted or excluded from CommonDevice during conversion:

• OpenVPN TLS authentication keys - Excluded at the conversion layer, never enter the enrichment pipeline
• IPsec pre-shared keys - Nuanced handling by platform:
  • OPNsense: schema.IPsec.PreSharedKeys IS mapped to common.IPsecConfig.PreSharedKeys, but the OPNsense MVC model stores UUID references to the Ipsec/KeyPairs/PreSharedKey model (not raw key material), so no credential leak currently. If a future schema revision stores raw keys in this field, redaction logic must be added
  • pfSense: IPsecPhase1.PreSharedKey is a scalar raw key but is intentionally NOT mapped into common.IPsecPhase1Tunnel — see pkg/parser/pfsense/converter_services.go convertIPsecPhase1Tunnels() implementation and the TestConverter_IPsecPhase1_PreSharedKeyExclusion regression test
• WireGuard interface private keys - Only public keys are mapped; pre-shared keys are mapped but redacted by the pipeline

Statistics Redaction#

The redactStatisticsServiceDetails function handles a special case where the SNMP community string copied into Statistics.ServiceDetails during computation must be post-processed for redaction.

The function signature is:

func redactStatisticsServiceDetails(stats *common.Statistics) *common.Statistics

The function returns a pointer, not void. The call site in prepareForExport is:

cp.Statistics = redactStatisticsServiceDetails(cp.Statistics)

The function is non-mutating: it returns the input pointer unchanged when no redaction is needed, or clones the Statistics struct and affected ServiceDetails when SNMP community redaction is required:

if len(matches) == 0 {
    return stats // No SNMP community to redact
}

out := *stats // Clone Statistics struct
out.ServiceDetails = slices.Clone(stats.ServiceDetails) // Clone slice
for _, idx := range matches {
    out.ServiceDetails[idx].Details = maps.Clone(stats.ServiceDetails[idx].Details) // Clone map
    out.ServiceDetails[idx].Details["community"] = redactedValue
}
return &out

This non-mutating design allows EnrichForExport to safely share a Statistics pointer across mixed redact=true and redact=false callers without leaking redacted values. A device enriched once can be exported multiple times with different redaction settings; the redacted export receives a cloned Statistics with redacted ServiceDetails, while the original device retains the unredacted Statistics for subsequent non-redacted exports.

The function redacts every matching SNMP entry, not just the first. analysis.ComputeStatistics emits a single SNMP entry today, but a future schema change (SNMPv3, separate read/write communities, multi-instance agents) could surface multiple — the redactor processes all of them to avoid leaving any in cleartext.

Statistics and Analysis Functions#

ComputeStatistics#

The analysis.ComputeStatistics function analyzes the device configuration and returns aggregated metrics organized into several categories:

Network Infrastructure Statistics:

• Interface counts by type (physical, VLAN, bridge, PPP, etc.)
• IPv4/IPv6 presence per interface
• VLAN, bridge, gateway, gateway group counts

Firewall Statistics:

• Total firewall rule count
• Rules by interface and type (pass/block/reject)
• NAT mode and entry counts

Services Statistics:

• DHCP scope counts with range details (covers both ISC and Kea scopes, which are unified in the same DHCP slice; HasDHCP() returns len(d.DHCP) > 0)
• Enabled services (DHCP, Unbound DNS, SNMP, SSH, NTP)
• Service configuration details (ports, bind addresses, community strings)

Security Features:

• Block private/bogon network flags
• HTTPS WebGUI status
• NAT reflection configuration

Summary Metrics:

• Total configuration item count
• Security score (0-100, computed by analysis.ComputeSecurityScore)
• Configuration complexity score (0-100, computed by analysis.ComputeConfigComplexity)

ComputeAnalysis#

The analysis.ComputeAnalysis function performs lightweight configuration analysis, categorizing findings into five types:

Dead Rule Detection (analysis.DetectDeadRules):

• Identifies shadowed firewall rules (rules blocked by earlier matching rules)
• Detects disabled rules that remain in the configuration
• Each finding includes a structured Kind field ("unreachable" or "duplicate") for programmatic classification
• Uses analysis.RulesEquivalent to detect rule overlaps

Unused Interface Detection (analysis.DetectUnusedInterfaces):

• Interfaces with no firewall rules
• Interfaces with no DHCP scopes
• Interfaces not referenced by NAT or VPN configurations

Security Issues (analysis.DetectSecurityIssues):

• HTTP-only WebGUI (missing HTTPS)
• Private/bogon networks not blocked
• Missing NAT reflection
• Weak or missing security configurations

Performance Issues (analysis.DetectPerformanceIssues):

• Excessive firewall rule counts
• Complex NAT configurations
• Missing gateway monitoring

Consistency Issues (analysis.DetectConsistency):

• Firewall rules referencing non-existent interfaces
• Gateway groups with missing gateways
• DHCP scopes on disabled interfaces
• Invalid gateway IP address formats (validated using net.ParseIP)

Integration with Format Exporters#

Generator Interface Architecture#

The enrichment pipeline integrates with format exporters through a two-tier interface hierarchy:

type Generator interface {
    Generate(ctx context.Context, cfg *common.CommonDevice, opts Options) (string, error)
}

type StreamingGenerator interface {
    Generator
    GenerateToWriter(ctx context.Context, w io.Writer, cfg *common.CommonDevice, opts Options) error
}

The base Generator interface produces output as a complete string, useful when further processing is needed. The StreamingGenerator interface extends this with incremental output via io.Writer, enabling memory-efficient generation for large configurations.

Format-Specific Usage#

Different export formats interact with the enrichment pipeline in distinct ways:

JSON Exporter (generateJSON):

1. Calls prepareForExport(data, redact)
2. Marshals enriched device to JSON with indentation
3. All enrichment fields and the pass-through ComplianceResults field appear in JSON output

YAML Exporter (generateYAML):

1. Identical pattern—calls prepareForExport(data, redact)
2. Marshals enriched device to YAML
3. Uses same enrichment data as JSON, including ComplianceResults

Markdown Exporter (generateMarkdown):

• Calls prepareForExport to enrich the device and apply optional redaction
• Uses builder pattern to generate standard or comprehensive reports
• When device.ComplianceResults is present, appends an audit section via BuildAuditSection() to include compliance findings
• Operates on both enriched statistics/analysis fields and the pass-through ComplianceResults data for comprehensive documentation

All exporters benefit from the same enrichment data (statistics, analysis, security assessments) while the ComplianceResults field, when populated by the audit handler, flows through to all formats without modification by the enrichment system.

Redact Parameter Flow#

The redact parameter is defined in Options and flows through the export pipeline:

type Options struct {
    // Redact controls whether sensitive fields are replaced with [REDACTED]
    Redact bool
    // ... other options
}

The flow proceeds: CLI flag → Options.Redact → format dispatcher → prepareForExport(data, opts.Redact) → conditional redaction application. This design makes redaction opt-in at the CLI level while ensuring the mechanism is consistently applied across all structured formats.

Critical Implementation Gotchas#

1. Shared Analysis Package (Resolved Circular Dependency)#

Prior to PR #409, the enrichment pipeline could not import internal/processor due to circular dependency constraints, leading to mirrored analysis logic between the converter and processor packages. This architectural limitation has been resolved by extracting shared analysis functions into internal/analysis/.

Previous Constraint:

• internal/converter/enrichment.go and internal/processor/processor.go both imported pkg/model
• Direct import between converter and processor would create circular dependency
• Required duplicating analysis logic in both packages

Solution (PR #409):

• Created internal/analysis/ package containing shared detection, statistics, and rule comparison logic
• Both converter and processor now import internal/analysis/ to access common functionality
• Eliminated code duplication while maintaining clean architecture

Shared Analysis Functions:

• analysis.ComputeStatistics() - Aggregates configuration metrics
• analysis.ComputeAnalysis() - Performs configuration analysis
• analysis.DetectDeadRules() - Identifies unreachable and duplicate firewall rules
• analysis.DetectUnusedInterfaces() - Finds enabled but unused interfaces
• analysis.DetectSecurityIssues() - Detects security configuration problems
• analysis.DetectPerformanceIssues() - Identifies performance concerns
• analysis.DetectConsistency() - Validates configuration consistency
• analysis.RulesEquivalent() - Compares firewall rules for functional equivalence

The shared analysis package maintains the "no circular dependencies" architectural strength while eliminating the need for mirrored logic.

2. Pointer Types for Enrichment Fields#

Enrichment fields must use pointer types in the CommonDevice structure:

Statistics *Statistics // Pointer allows nil checks
Analysis *Analysis // nil = "not yet computed"
SecurityAssessment *SecurityAssessment // Distinguishes from "computed but empty"

Rationale:

• Distinguishes "not computed" (nil) from "computed but empty" (non-nil with zero values)
• Export functions check if device.Statistics != nil before including fields
• Memory efficiency—large structures only allocated when needed
• Enables lazy computation—enrichment only occurs when needed

This pattern is critical for the pipeline's efficiency, as it avoids computing expensive statistics for operations that don't require them.

3. Immutability Requirements#

The original CommonDevice must never be mutated by the enrichment pipeline. This immutability guarantee is a foundational design principle.

Problem: Shallow copy of slice fields creates shared references—modifying elements affects the original

Solution: Selective deep-copying before modification:

// WRONG - modifies caller's data
cp.Certificates[0].PrivateKey = "[REDACTED]"

// CORRECT - deep copy before redaction
cp.Certificates = slices.Clone(cp.Certificates)
cp.Certificates[0].PrivateKey = "[REDACTED]"

Benefit: Multiple exports with different redaction settings from the same source data

This design enables use cases like exporting both redacted and unredacted versions of a configuration without re-parsing the XML.

4. PR #326 Mutation Bug Fix#

Issue: Earlier implementations mutated the input CommonDevice in-place, permanently redacting the original device after a single export. This was a critical bug that violated the immutability contract.

Root Cause: Redaction was applied directly to nested structures without cloning:

// Old buggy code
func prepareForExport(device *common.CommonDevice, redact bool) {
    if redact {
        device.Certificates[i].PrivateKey = "[REDACTED]" // MUTATES ORIGINAL
    }
}

Fix (PR #315, PR #326):

• Added defensive deep-copying using slices.Clone() before mutation
• Changed to return new copy rather than modifying input
• Tested to verify original data remains unmodified

// Fixed code
if len(cp.Certificates) > 0 {
    cp.Certificates = slices.Clone(cp.Certificates) // COPY FIRST
    for i := range cp.Certificates {
        if cp.Certificates[i].PrivateKey != "" {
            cp.Certificates[i].PrivateKey = redactedValue
        }
    }
}

This fix demonstrates the importance of defensive copying when working with nested data structures in Go.

5. ComplianceResults is Populated Externally#

The ComplianceResults field has a unique data flow that differs from other enrichment fields:

Standard Enrichment Fields (Statistics, Analysis, SecurityAssessment, PerformanceMetrics):

1. Nil when device enters prepareForExport()
2. Computed by prepareForExport() if not already present
3. Populated during export preparation

ComplianceResults Field:

1. Populated BEFORE enrichment by mapAuditReportToComplianceResults() in cmd/audit_handler.go
2. Passed through prepareForExport() unchanged (neither computed nor modified)
3. Available to all export formats (JSON, YAML, Markdown) without enrichment layer understanding audit types

Rationale:

• Compliance data comes from the audit package, which the converter package cannot import due to circular dependency constraints
• Mapping audit results to the common model (ComplianceResults) occurs in the cmd layer, which can import both audit and converter
• This architecture keeps enrichment focused on generic device data analysis, not audit-specific concerns
• The ComplianceResults type mirrors audit.Report structure but lives in the pkg/model package to avoid circular dependencies

Type Information:

• Type: *ComplianceResults (not the old stub ComplianceChecks struct)
• Contains: ComplianceFinding, PluginComplianceResult, ComplianceControl, ComplianceResultSummary
• Mirrors audit.Report structure with per-plugin findings, controls, and aggregate summaries

6. New Enrichment Fields Must Be Wired#

When adding new enrichment fields to CommonDevice, developers must follow a specific integration pattern to ensure the fields appear in exports:

1. Define the field as a pointer type in device.go
2. Add computation logic in the appropriate internal/analysis/ file
3. Wire the field in the enrich() helper to populate it before export (this helper backs both prepareForExport and EnrichForExport)
4. Add JSON/YAML struct tags with omitempty directive

Gotcha: Fields not wired in enrich() will never appear in JSON/YAML output, even if they're defined in the struct and have computation logic implemented elsewhere. Developers adding new enrichment fields must ensure the field is correctly nil-checked and populated in the shared enrich() helper so both the single-format path (prepareForExport) and the multi-format optimization path (EnrichForExport) benefit. Note that ComplianceResults is an exception to this rule—it is populated externally by the audit handler and passes through enrichment unchanged.

7. SNMP Type Name Difference#

The SNMP field in CommonDevice uses type SNMPConfig, not SNMP:

SNMP SNMPConfig `json:"snmp,omitempty" yaml:"snmp,omitempty"`

This naming pattern (field name SNMP, type name SNMPConfig) applies to several configuration subsystems to avoid naming conflicts with packages or functions. This is a common Go idiom but can be confusing for developers unfamiliar with the codebase.

Relevant Code Files#

Related Topics#

• CommonDevice Data Model - The platform-agnostic configuration model that serves as the central data structure for the enrichment pipeline
• Converter Architecture - The broader conversion system including parsers, converters, and format generators that provides the context for enrichment
• Audit Processing Pipeline - The separate processor package for interactive analysis and markdown reports, which now uses the shared internal/analysis package
• Sensitive Data Redaction - Security patterns and best practices for protecting credentials in exported configurations
• Configuration Analysis - Automated detection of dead rules (with structured Kind classification), unused interfaces, security issues, and consistency problems in firewall configurations
• Statistics Computation - Aggregation of configuration metrics for reporting, dashboards, and compliance monitoring