Multi-Format Export System#

Overview#

The Multi-Format Export system in opnDossier is a comprehensive data transformation and output generation architecture that converts OPNsense firewall configurations into five distinct output formats: Markdown, JSON, YAML, plain text, and HTML. The system operates through a unified data pipeline built around a platform-agnostic data model, providing consistent representation across all formats while supporting format-specific optimizations.

At its core, the system employs the HybridGenerator orchestrator to dispatch format-specific conversion operations. All exports pass through an enrichment pipeline that populates computed fields including statistics, security assessments, configuration analysis, and performance metrics. The architecture supports optional redaction of sensitive data (passwords, private keys, API secrets, SNMP community strings) while preserving accurate analysis results.

The system is designed for offline operation in security-sensitive environments, with streaming interfaces for memory-efficient processing of large configurations, and auto-naming capabilities for batch file processing.

Core Architecture#

Single Source of Truth: CommonDevice Model#

All export formats derive from the CommonDevice struct, which provides a platform-agnostic representation of firewall configurations. This model acts as a normalized intermediate representation between vendor-specific XML schemas and output formats.

Domain organization:

• System: Hostname, web GUI configuration, SSH settings, users, groups
• Network: Interfaces, VLANs, bridges, routing tables, gateways
• Security: Firewall rules, NAT configuration, IDS/IPS settings, VPN (OpenVPN, WireGuard, IPsec)
• Services: DHCP, DNS (Unbound), NTP, SNMP, syslog, load balancing, high availability

The model includes enrichment fields populated during export:

• Statistics - Aggregated counts, service detection, security features
• Analysis - Configuration issues (dead rules, unused interfaces, security warnings)
• SecurityAssessment - Security scoring and feature inventory
• PerformanceMetrics - Configuration complexity scoring

HybridGenerator: Unified Orchestrator#

The HybridGenerator implements both the Generator and StreamingGenerator interfaces, providing two generation modes:

1. String-based generation (Generate()) - Returns complete output as a string, useful for post-processing (e.g., HTML conversion from markdown)
2. Streaming generation (GenerateToWriter()) - Writes directly to io.Writer for memory efficiency

The generator dispatches to format-specific converters through the DefaultRegistry, which maps format names and aliases to FormatHandler implementations:

• markdown: Programmatic builder pattern
• json: Direct struct serialization
• yaml: Direct struct serialization
• text: Markdown → HTML → plain text pipeline
• html: Markdown → HTML with CSS styling

The DefaultRegistry serves as the single source of truth for supported formats, format validation, file extension mapping, and handler dispatch. Adding a new format requires only implementing the FormatHandler interface and registering it in newDefaultRegistry().

Format Registry#

The FormatRegistry centralizes format dispatch by mapping canonical format names and aliases to FormatHandler implementations. It replaces scattered switch statements with a single source of truth for supported formats.

Architecture:

• DefaultRegistry is the package-level singleton pre-populated with all built-in format handlers
• Get(format) returns the handler for a format name (canonical or alias), returning ErrUnsupportedFormat if not found
• Canonical(format) resolves aliases to canonical format names (e.g., md → markdown, yml → yaml)
• ValidFormats() returns a sorted slice of canonical format names
• Extensions() returns a map of canonical format names to file extensions
• Register(format, handler) adds a new format handler, panicking on duplicate registration (following the database/sql driver pattern)

FormatHandler Interface:
Each format implements the FormatHandler interface:

• FileExtension() - Returns the file extension (e.g., .md, .json)
• Aliases() - Returns alternative names for the format (e.g., ["md"] for markdown)
• Generate(g, data, opts) - Produces output as a string
• GenerateToWriter(g, w, data, opts) - Streams output directly to an io.Writer

Benefits:

• Adding a new format requires only implementing FormatHandler and registering it in newDefaultRegistry()
• Format validation is centralized via DefaultRegistry.Get()
• File extension determination uses handler.FileExtension() from the registry
• Alias resolution is automatic (e.g., md, yml, txt, htm work seamlessly)

Supported Export Formats#

1. Markdown (.md)#

Implementation: Programmatic builder pattern using the MarkdownBuilder interface.

Architecture:

• Uses github.com/nao1215/markdown library for structured generation
• Builds four major sections: System, Network, Security, Services
• Supports terminal rendering via Glamour with theme support (auto/dark/light)
• Implements the SectionWriter interface for true section-by-section streaming

Content sections:

• Basic info, WebGUI, sysctl, users, groups
• Interface details with hyperlinks
• Firewall rules tables
• IDS/Suricata configuration
• DHCP, DNS, SNMP, NTP, load balancer services

Advantages: Type-safe, compile-time validated, human-readable terminal output.

Usage:

opndossier convert config.xml -f markdown -o report.md
opndossier convert config.xml -f md # alias

2. JSON (.json)#

Implementation: Direct struct serialization.

Architecture:

• Uses Go's standard json.MarshalIndent with 2-space indentation
• Calls prepareForExport() for enrichment and optional redaction
• Streaming variant uses json.NewEncoder for direct writing

Enrichment fields included:

• statistics - Interface counts, firewall rules, NAT, services, security features
• analysis - Dead rules, unused interfaces, security/performance/consistency issues
• securityAssessment - Overall security score, enabled features
• performanceMetrics - Configuration complexity score

Advantages: Zero custom logic, programmatic access, integration with JSON tools (jq, etc.).

Usage:

opndossier convert config.xml -f json -o config.json
opndossier convert config.xml --redact -f json # with sensitive data redaction

3. YAML (.yaml, .yml)#

Implementation: Parallel to JSON with YAML serialization.

Architecture:

• Uses gopkg.in/yaml.v3 for marshaling
• Same prepareForExport() preprocessing as JSON
• Streaming uses yaml.NewEncoder with 2-space indentation

Advantages: Human-readable alternative to JSON, configuration management integration (Ansible, Terraform).

Usage:

opndossier convert config.xml -f yaml -o config.yaml
opndossier convert config.xml -f yml # alias

4. Plain Text (.txt)#

Implementation: Derived format via markdown → HTML → text pipeline.

Architecture:

• Three-stage pipeline:
  1. Renders markdown to HTML via goldmark renderer
  2. Extracts tables and alerts with placeholders
  3. Converts HTML to text via html2text

Format conversions:

• Tables preserved as tab-separated values
• Alerts converted to TYPE:\ntext format
• Links converted to text (url) format

Advantages: Maintains readability without formatting, suitable for basic text editors.

Usage:

opndossier convert config.xml -f text -o report.txt
opndossier convert config.xml -f txt # alias

5. HTML (.html, .htm)#

Implementation: Derived format via markdown → HTML pipeline.

Architecture:

• Uses goldmark renderer to convert markdown to HTML
• Wraps output in self-contained HTML shell with embedded CSS
• Transforms GitHub-style alert blockquotes ([!NOTE], [!WARNING], etc.) into styled divs
• Supports dark/light mode via CSS media queries

Advantages: Consistent with markdown output, offline-viewable, no external dependencies.

Usage:

opndossier convert config.xml -f html -o report.html
opndossier convert config.xml -f htm # alias

Optimizing Multi-Format Exports with EnrichForExport#

EnrichForExport() Function#

The EnrichForExport() function is the explicit memoization entry point for callers preparing the same device for multiple format exports (e.g., JSON + YAML + Markdown). The computeStatistics and computeAnalysis functions are O(n) over interfaces, rules, and services, and dominate per-format export time. Calling EnrichForExport once before the format loop avoids recomputing them per format.

Usage pattern:

device := parseDevice(xmlData)
EnrichForExport(device) // Populate Statistics, Analysis, etc. once

// Each format export inherits the cached values
mdOutput, _ := generator.Generate(ctx, device, Options{Format: FormatMarkdown})
jsonOutput, _ := generator.Generate(ctx, device, Options{Format: FormatJSON})
yamlOutput, _ := generator.Generate(ctx, device, Options{Format: FormatYAML})

What it populates:

• DeviceType (defaulting to OPNsense)
• Statistics - Aggregated counts, service detection, security features
• Analysis - Configuration issues (dead rules, unused interfaces, security warnings)
• SecurityAssessment - Security scoring and feature inventory
• PerformanceMetrics - Configuration complexity scoring

Performance: Benchmark results on sample.config.6.xml (~119KB) show a ~65% reduction in latency and allocations for the bare prepareForExport call when using EnrichForExport. The realistic multi-format CLI workload (markdown + JSON + YAML) sees a ~7.7% latency reduction as serialization dominates the headline benchmark.

SECURITY WARNING: EnrichForExport does NOT redact sensitive fields. The resulting *CommonDevice carries plaintext secrets—most notably the SNMP community string in Statistics.ServiceDetails—because analysis functions must observe unredacted input for accurate presence checks. Callers MUST NOT marshal or log the device directly after calling EnrichForExport. Always pass through prepareForExport (or a Generator that calls prepareForExport) so the redact branch can produce a clone with sensitive fields stripped.

CACHE INVALIDATION: EnrichForExport memoizes Statistics and Analysis as a snapshot of the device at call time. If the caller mutates a field that feeds those computations (e.g., device.SNMP.ROCommunity, FirewallRules) after calling EnrichForExport, the cached values go stale and subsequent exports will reflect the pre-mutation state. Re-call EnrichForExport (after first clearing the affected enrichment field) when the underlying configuration changes between exports.

Export Enrichment Pipeline#

prepareForExport() Function#

The prepareForExport() function serves as the gateway for JSON and YAML exports, performing enrichment and optional redaction:

Flow:

1. Shallow Copy Creation: cp := *data creates a shallow copy to avoid mutating the original device. Configuration data is shared (read-only), but enrichment fields are independent pointers.
2. Enrichment: Calls enrich() to populate DeviceType, Statistics, Analysis, SecurityAssessment, and PerformanceMetrics when nil. The enrich() call runs before redaction so analysis functions observe unredacted input for presence checks.
3. Redaction: When redact=true, calls redactSensitiveFields() to deep-copy slices and replace sensitive values with [REDACTED], then calls redactStatisticsServiceDetails() to post-process computed statistics.

Memoization: Repeated calls to prepareForExport on the same device will skip the expensive computeStatistics and computeAnalysis work if EnrichForExport was called first. The enrich() function checks whether enrichment fields are nil before populating them, so pre-populated fields are reused.

computeStatistics()#

The computeStatistics() function aggregates configuration data into structured statistics:

Interface Statistics:

• Total interface count
• Interfaces grouped by type (physical, VLAN, bridge, etc.)
• Per-interface IPv4/IPv6 configuration, DHCP status, security settings (BlockPrivate, BlockBogons)

Network Infrastructure:

• VLAN count, bridge count, certificate count, CA count

Firewall Statistics:

• Total rules
• Rules grouped by interface
• Rules grouped by type (pass, block, reject)

NAT Configuration:

• NAT mode (automatic/hybrid/advanced/disabled)
• Total NAT entries (inbound + outbound)

Service Detection:
Automatically detects and catalogs enabled services:

• DHCP Server (per-interface with range information)
• Unbound DNS Resolver
• SNMP Daemon (with community string details)
• SSH Daemon (with group details)
• NTP Daemon (with preferred server)

Security Features:
Identifies enabled security features:

• Block Private Networks (RFC 1918 blocking)
• Block Bogon Networks
• HTTPS Web GUI
• NAT Reflection Disabled

Summary Metrics:

• TotalConfigItems: Sum of all configuration components
• SecurityScore (0-100): Security features (10pts each) + firewall rules (20pts) + HTTPS GUI (15pts) + SSH config (10pts) + IDS/IPS (15-25pts)
• ConfigComplexity (0-100): Normalized weighted sum of configuration elements

computeAnalysis()#

The computeAnalysis() function performs five types of analysis:

Dead Rule Detection:

• Unreachable rules (following block-all rule)
• Duplicate rules on the same interface
• Recommendations for removal or reordering

Unused Interface Detection:

• Identifies enabled interfaces not used in firewall rules, DHCP, DNS, VPN, or load balancing

Security Issues:

• Insecure Web GUI (HTTP instead of HTTPS) - critical severity
• Default SNMP Community ("public") - high severity
• Overly Permissive WAN Rules (source=any) - high severity

Performance Issues:

• Checksum Offloading Disabled - low severity
• Segmentation Offloading Disabled - low severity
• High Rule Count (>500 rules) - medium severity

Consistency Issues:

• Invalid Gateway Format - medium severity
• DHCP Without Interface IP - high severity
• User-Group Mismatch - medium severity

Redaction Support#

Sensitive Field Redaction#

The redactSensitiveFields() function replaces sensitive values with [REDACTED]:

1. High Availability Password: HighAvailability.Password
2. Certificate Private Keys: Certificates[].PrivateKey - Deep-copied via make + copy before mutation during JSON/YAML serialization via redactedCopyUnsafe() in internal/processor/report.go
3. CA Private Keys: CAs[].PrivateKey - Deep-copied via make + copy before mutation during JSON/YAML serialization via redactedCopyUnsafe() in internal/processor/report.go
4. API Key Secrets: Users[].APIKeys[].Secret
5. SNMP Community String: SNMP.ROCommunity
6. WireGuard Pre-Shared Keys: VPN.WireGuard.Clients[].PSK
7. DHCPv6 Authentication Secrets: DHCP[].AdvDHCP6KeyInfoStatementSecret

Implementation details:

• Redaction is conditional — only entries with non-empty sensitive values are redacted. Empty PrivateKey fields remain empty to avoid injecting [REDACTED] where no data existed.
• The implementation uses helper functions hasCertPrivateKeys() and hasCAPrivateKeys() to gate redaction logic, ensuring efficient processing by checking for presence of sensitive data before deep-copying slices.
• Certificate and CA redaction is documented in detail in AGENTS.md §5.25.

Note: OpenVPN TLS keys, IPsec pre-shared keys, and WireGuard private keys are never included in the CommonDevice model (excluded at the converter mapping layer).

Statistics Redaction#

The redactStatisticsServiceDetails() function returns a *common.Statistics whose sensitive ServiceDetails values are replaced with the redaction marker. The input is not mutated: when redaction is required, the function clones the Statistics struct, the ServiceDetails slice, and the affected per-element Details map. When no SNMP community redaction is needed, the input pointer is returned unchanged.

Non-mutating design:

func redactStatisticsServiceDetails(stats *common.Statistics) *common.Statistics {
    // Find all SNMP entries with "community" key
    var matches []int
    for i := range stats.ServiceDetails {
        if stats.ServiceDetails[i].Name == analysis.ServiceNameSNMP &&
           stats.ServiceDetails[i].Details != nil {
            if _, ok := stats.ServiceDetails[i].Details["community"]; ok {
                matches = append(matches, i)
            }
        }
    }
    if len(matches) == 0 {
        return stats // No redaction needed, return input unchanged
    }

    // Clone and redact
    out := *stats
    out.ServiceDetails = slices.Clone(stats.ServiceDetails)
    for _, idx := range matches {
        out.ServiceDetails[idx].Details = maps.Clone(stats.ServiceDetails[idx].Details)
        out.ServiceDetails[idx].Details["community"] = redactedValue
    }
    return &out
}

This non-mutating design allows EnrichForExport to safely share a single Statistics pointer across mixed redact=true and redact=false callers without leaking redaction markers back into the caller's data. The function also redacts all matching SNMP community entries, not just the first—a defense-in-depth measure for future schema changes (SNMPv3, separate read/write communities, multi-instance agents) that might surface multiple SNMP services.

CLI Flag#

Redaction is controlled by the --redact flag:

opndossier convert --redact -f json config.xml -o config.json
opndossier convert --redact -f yaml config.xml -o config.yaml

The flag applies only to JSON and YAML formats. Markdown, HTML, and plain text formats do not include sensitive fields in their output.

Auto-Naming for Multiple Files#

Multiple File Processing#

When processing multiple input files, the system automatically generates output filenames based on input filenames with appropriate extensions:

# Process multiple files (each gets auto-named output)
opndossier convert config1.xml config2.xml config3.xml

# Convert multiple files to JSON
opndossier convert -f json config1.xml config2.xml config3.xml

Auto-naming rules:

• Each output file is named based on its input file
• Format-specific extensions are applied automatically: .md, .json, .yaml, .txt, .html
• The --output flag is ignored when processing multiple files
• Input filename: firewall-config.xml → Output: firewall-config.json (for JSON format)

Batch Processing Examples#

Process all XML files in a directory:

for file in *.xml; do
    opndossier convert "$file" -f json -o "${file%.xml}.json"
done

Parallel processing:

find . -name "*.xml" | xargs -P 4 -I {} opndossier convert {} -f yaml

Streaming Interfaces#

StreamingGenerator Interface#

The StreamingGenerator interface extends the base Generator interface with streaming support:

Required methods:

• Generate(ctx context.Context, cfg *CommonDevice, opts Options) (string, error) - Returns complete output as string
• GenerateToWriter(ctx context.Context, w io.Writer, cfg *CommonDevice, opts Options) error - Writes directly to io.Writer

GenerateToWriter Implementation#

The GenerateToWriter method validates input, retrieves the appropriate handler from DefaultRegistry, and dispatches to format-specific writers:

• Markdown: generateMarkdownToWriter - True streaming via SectionWriter interface
• JSON: generateJSONToWriter - Direct encoding with json.NewEncoder(w)
• YAML: generateYAMLToWriter - Direct encoding with yaml.NewEncoder(w)
• Text: generatePlainTextToWriter - Generates markdown, strips formatting
• HTML: generateHTMLToWriter - Generates markdown, converts to HTML

Format dispatch uses the handlerForFormat() helper, which retrieves handlers from DefaultRegistry and defaults to markdown when the format is empty.

SectionWriter Interface#

For true section-by-section streaming, the SectionWriter interface defines methods for writing individual sections:

• WriteSystemSection(w io.Writer, data *CommonDevice) error
• WriteNetworkSection(w io.Writer, data *CommonDevice) error
• WriteSecuritySection(w io.Writer, data *CommonDevice) error
• WriteServicesSection(w io.Writer, data *CommonDevice) error
• WriteStandardReport(w io.Writer, data *CommonDevice) error
• WriteComprehensiveReport(w io.Writer, data *CommonDevice) error

The MarkdownBuilder implements this interface, reducing peak memory usage for large configurations.

Streaming Format Support#

Note: Markdown provides the best streaming benefits as sections are written incrementally. Other formats require full document serialization or post-processing.

Usage Examples#

Single File Export#

# Export to all formats
opndossier convert config.xml -o report.md
opndossier convert config.xml -f json -o config.json
opndossier convert config.xml -f yaml -o config.yaml
opndossier convert config.xml -f text -o report.txt
opndossier convert config.xml -f html -o report.html

With Redaction#

# Redact sensitive data in JSON/YAML exports
opndossier convert --redact -f json config.xml -o config-redacted.json
opndossier convert --redact -f yaml config.xml -o config-redacted.yaml

Batch Processing#

# Process all XML files
for file in *.xml; do
    opndossier convert "$file" -f json -o "${file%.xml}.json"
done

# Parallel processing (4 workers)
find . -name "*.xml" | xargs -P 4 -I {} opndossier convert {} -f yaml

Configuration Analysis#

# Extract firewall rules with jq
opndossier convert config.xml -f json | jq '.filter.rule[]'

# Count rules by type
opndossier convert config.xml -f json | jq '.filter.rule | group_by(.type) | map({type: .[0].type, count: length})'

# Compare configurations
opndossier convert current.xml -f json | jq -S . > current.json
opndossier convert previous.xml -f json | jq -S . > previous.json
diff current.json previous.json

CI/CD Integration#

# .github/workflows/documentation.yml
name: Generate Network Documentation

on:
  push:
    paths:
      - configs/*.xml

jobs:
  generate-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.26'
      - name: Install opnDossier
        run: go install github.com/EvilBit-Labs/opnDossier@latest
      - name: Generate Reports
        run: |
          mkdir -p reports
          for config in configs/*.xml; do
            filename=$(basename "$config" .xml)
            opndossier convert "$config" -f json -o "reports/${filename}.json"
            opndossier convert "$config" -f html -o "reports/${filename}.html"
          done

Relevant Code Files#

Related Topics#

Configuration Validation#

See the validate command for XML schema validation and configuration consistency checks before export.

Terminal Display#

See the display command for interactive terminal rendering with syntax highlighting and color themes.

Audit Mode#

See Audit Mode documentation for compliance reporting with STIG, SANS, and firewall-specific plugins.

Custom Templates#

Future enhancement: Support for custom export templates and format plugins.

References#

• opnDossier GitHub Repository
• Architecture Documentation
• Usage Guide
• Configuration Reference