Package Restructuring Roadmap#

The Package Restructuring Roadmap was a strategic architectural evolution plan for the opnDossier project that transitioned the codebase from an internal re-export architecture to a clean, public-facing package structure. Initiated in Issue #301 (v1.3.0) and Issue #302 (v2.0.0), the restructuring was completed by PR #404, improving extensibility, reducing maintenance overhead, and enabling external Go projects to integrate with opnDossier's parsing and data model infrastructure.

The restructuring accomplished two primary goals: (1) relocated types from internal/ to pkg/ to expose schemas and the platform-agnostic CommonDevice model as importable public APIs, and (2) eliminated the 92 type aliases in the internal/model/ re-export layer that previously provided convenience imports. The migration updated 173 files across the codebase, resulting in a net deletion of 1,303 lines by removing the entire re-export shim layer. The restructuring follows Go best practices for public API design and prepares the codebase for multi-device support (pfSense, Cisco ASA, and future platforms).

Architecture Evolution#

Layered Model Design#

The opnDossier architecture implements a four-layer data model that cleanly separates XML parsing concerns from platform-agnostic domain logic. After PR #404, the architecture has been restructured as follows:

1. pkg/schema/opnsense/ — XML data transfer objects (DTOs) with xml:"" tags mirroring device-specific configuration structures
2. pkg/parser/opnsense/ — Device-specific parser and converter layer (only package importing pkg/schema/opnsense/)
3. pkg/model/ — Platform-agnostic CommonDevice domain model with no XML tags
4. pkg/parser/ — Factory and DeviceParser interface for device type detection

This layered design ensures that downstream consumers (CLI commands, converters, audit plugins, processors) operate exclusively on the platform-agnostic CommonDevice model, while device-specific XML parsing remains isolated in dedicated parser packages.

Re-Export Layer Pattern (Eliminated)#

The internal/model/ package previously served as a convenience re-export layer containing 92 type aliases distributed across 13 files. This pattern, while convenient initially, created maintenance overhead and obscured the actual data model organization. Issue #282 documented the rationale for eliminating this layer, noting that synchronizing 92 type aliases across 13 files with their underlying implementations increased cognitive load and complicated IDE navigation.

PR #404 completely removed the re-export layer, along with all 92 type aliases and constructor wrappers. External projects now import types directly from pkg/model/, pkg/schema/opnsense/, and pkg/parser/.

Factory Pattern#

The Factory (renamed from ParserFactory to follow Go naming conventions) implements device type detection and parser delegation through two methods:

• createWithAutoDetect() — Inspects XML root element to identify device type using peekRootElementBounded() with context-aware cancellation
• createWithOverride() — Bypasses auto-detection when --device-type flag is specified

The factory is instantiated with an OPNsenseXMLDecoder interface (satisfied by cfgparser.NewXMLParser()) to keep pkg/ packages free of internal/ dependencies:

factory := parser.NewFactory(cfgparser.NewXMLParser())
device, warnings, err := factory.CreateDevice(ctx, reader, "", false)

The implementation uses a DeviceParserRegistry for device type selection, enabling pluggable parser implementations as implemented in PR #437.

Motivation for Restructuring#

The current architecture presents several limitations that motivate the restructuring:

1. Go visibility constraints — internal/ packages cannot be imported by external Go projects, preventing third-party integrations
2. Maintenance overhead — 92 type aliases require constant synchronization with underlying types
3. Code navigation issues — IDE autocomplete and code tracing complicated by multiple layers of indirection
4. Extensibility barriers — Hardcoded switch statements prevent pluggable parser implementations (addressed in PR #437)
5. Coupling — Factory directly depends on all device parser implementations, violating dependency inversion (addressed in PR #437)

The restructuring addresses these limitations while maintaining opnDossier's five core design principles: offline-first (zero external dependencies), operator-focused (CLI-centric workflows), framework-first (leveraging established Go libraries), structured data (preserving configuration relationships), and security-first (no telemetry, strict validation).

Milestone v1.3.0: Public Package Migration (COMPLETED)#

Overview#

Issue #301 moved device schemas and the CommonDevice model from internal/ to pkg/ packages, enabling external Go projects to import opnDossier's data types and parsing pipeline. This milestone was completed by PR #404, which also eliminated the re-export layer originally planned for removal in v2.0.0.

Implemented Package Structure#

The restructuring introduced a new pkg/ directory hierarchy that mirrors the internal layering while exposing only stable, documented APIs:

pkg/
├── model/ # Platform-agnostic domain model
│ └── (from internal/model/common/)
│ CommonDevice, FirewallRule, Interface, VPN, DHCP, warning.go, etc.
│
├── schema/
│ └── opnsense/ # OPNsense XML DTO types
│ └── (from internal/schema/)
│ OpnSenseDocument, Firewall, Rule, etc.
│
└── parser/ # Parser interfaces + factory
    │ DeviceParser interface, Factory (formerly ParserFactory)
    │ (from internal/model/factory.go)
    │
    └── opnsense/ # OPNsense-specific implementation
        └── (from internal/model/opnsense/)
        Parser, Converter, NewParser(), NewConverter()

This structure maintains clear separation of concerns: pkg/model/ contains device-agnostic types consumed by all downstream code, pkg/schema/opnsense/ contains XML-specific DTOs used only by parser implementations, and pkg/parser/ defines the extensibility contracts for device type support.

Package Migration Mapping#

The following table documents the precise migration path implemented in PR #404:

The migration preserved file history through git mv and ensured attribution remains intact for all type definitions.

API Surface Stability#

All packages exposed under pkg/ commit to Semantic Versioning guarantees:

This stability commitment ensures external projects can depend on opnDossier's public APIs without unexpected breakage, following Go's module compatibility guidelines.

Package Boundary Enforcement#

PR #404's initial migration was mechanical (import path updates) but did not address structural dependencies. Four production files in pkg/ initially violated Go's internal/ access boundary by importing internal/cfgparser and internal/constants. These violations were resolved through two patterns documented in docs/solutions/architecture-issues/pkg-internal-import-boundary.md:

1. Constant extraction — pkg/schema/opnsense/constants.go defines NetworkAny locally, removing the internal/constants dependency
2. Interface injection — pkg/parser.OPNsenseXMLDecoder interface allows cmd/ layer to inject internal/cfgparser implementation without creating import dependencies

Boundary verification command:

grep -rn 'internal/' --include='*.go' pkg/ | grep -v _test.go

This command should be run before committing changes to pkg/ packages to catch boundary violations. The canonical example of interface injection is parser.NewFactory(cfgparser.NewXMLParser()), where the factory constructor accepts an OPNsenseXMLDecoder interface satisfied by the internal implementation. Go's structural typing allows pkg/parser/opnsense/ to define its own unexported xmlDecoder interface without importing internal/ packages.

Implementation Summary#

PR #404 implemented all four phases of the migration strategy:

Phase 1: Public Package Structure (Completed)#

Created the new pkg/ hierarchy and moved all files using git mv to preserve history:

• ✅ Created pkg/model/ from internal/model/common/
• ✅ Created pkg/schema/opnsense/ from internal/schema/
• ✅ Created pkg/parser/ with extracted DeviceParser interface
• ✅ Created pkg/parser/opnsense/ from internal/model/opnsense/
• ✅ Added pkg/model/warning.go for ConversionWarning types

Phase 2: Import Path Updates (Completed)#

Mechanical find-and-replace across 173 files updated import paths in all consuming subsystems:

Example transformation:

// Before
import "github.com/EvilBit-Labs/opnDossier/internal/model/common"

// After
import common "github.com/EvilBit-Labs/opnDossier/pkg/model"

Phase 3: Re-Export Layer Removal (Completed)#

PR #404 eliminated the backward compatibility bridge originally planned for v2.0.0:

• ✅ Deleted entire internal/model/ directory (17 files)
• ✅ Removed 92 type aliases from security.go, services.go, interfaces.go, system.go, vpn.go, and 8 other files
• ✅ Removed constructor wrapper functions
• ✅ No re-export shim remains — consumers import pkg/ directly

This immediate removal resulted in a breaking change, eliminating 1,303 lines of alias code.

Phase 4: Validation and Documentation (Completed)#

The final phase ensured correctness and communicated changes:

• ✅ CI checks pass with updated imports (just ci-check)
• ✅ All tests pass (unit, integration, E2E)
• ✅ Documentation updates:
  • AGENTS.md — AI agent instructions updated with new package structure
  • .github/copilot-instructions.md — GitHub Copilot guidance updated
  • .golangci.yml — Updated revive var-naming exclusion path to pkg/model/
  • Multiple documentation files updated with new import patterns
• ✅ Package boundary fixes:
  • Resolved four pkg/ → internal/ import violations through constant extraction and interface injection
  • Boundary verification command added: grep -rn 'internal/' --include='*.go' pkg/ | grep -v _test.go
  • Solution documented in docs/solutions/architecture-issues/pkg-internal-import-boundary.md

Impact Assessment#

The completed migration has the following impact profile:

• Files affected: 173 files with import path and structural changes
• Net change: +599 additions / -1,303 deletions (net -704 lines)
• Risk level: Low — purely mechanical refactoring with no logic changes
• Breaking change: Yes — internal/model/ re-export layer completely removed
• External consumption: Enabled — external Go projects can import pkg/model/, pkg/parser/, and pkg/schema/opnsense/

Milestone v2.0.0: Parser Registry Refactoring (✅ COMPLETED)#

Overview#

Issue #302 replaced the hardcoded switch dispatch in ParserFactory with a registry pattern, allowing external Go projects to register custom DeviceParser implementations for new device types (pfSense, Fortinet, MikroTik, etc.). This milestone transforms opnDossier from a single-device tool into an extensible multi-device platform. This milestone was completed by PR #437, which implemented a thread-safe DeviceParserRegistry following the database/sql driver registration pattern.

Dependencies: Requires completion of Issue #301 (v1.3.0) to expose public parser APIs that external implementations can satisfy.

Motivation#

While Issue #301 makes data types importable, Issue #302 enables external projects to produce CommonDevice instances by registering custom parsers that integrate with opnDossier's factory and auto-detection pipeline. This separation of concerns follows the Open/Closed Principle: opnDossier is open for extension (new device types) but closed for modification (no core code changes required).

Use case: A pfSense parser developed in a separate Go module registers itself at initialization, allowing opndossier convert pfsense-config.xml to work transparently without modifying opnDossier's codebase. The auto-detection mechanism inspects the XML root element (<pfsense> vs <opnsense>) and selects the appropriate registered parser.

Registry Pattern Design#

DeviceParserRegistry Structure#

The Factory uses a thread-safe DeviceParserRegistry for parser management (implemented in PR #437):

// pkg/parser/registry.go
type DeviceParserRegistry struct {
    mu sync.RWMutex
    parsers map[string]ConstructorFunc // rootElement → constructor
}

// ConstructorFunc is the factory function signature for creating DeviceParser
// instances. The OPNsenseXMLDecoder parameter allows injection of the XML parsing backend.
type ConstructorFunc = func(OPNsenseXMLDecoder) DeviceParser

// Register adds a parser constructor for a given XML root element.
// Panics on duplicate registration, nil factory, or empty device type.
func (r *DeviceParserRegistry) Register(deviceType string, fn ConstructorFunc)

// Get returns the constructor for the given device type, or (nil, false)
func (r *DeviceParserRegistry) Get(deviceType string) (ConstructorFunc, bool)

// List returns a sorted slice of all registered device type names.
func (r *DeviceParserRegistry) List() []string

The registry maps XML root element names (case-insensitive) to parser constructor functions, allowing dynamic parser instantiation based on configuration file inspection.

Thread Safety Guarantees#

The registry uses sync.RWMutex for thread-safe concurrent access:

• Register() acquires write lock during registration
• createWithAutoDetect() acquires read lock during lookup
• Registration must occur sequentially before concurrent factory access begins
• Go memory model guarantees ensure all writes in Register() are visible to subsequent readers

This design supports the common Go pattern of sequential initialization during init() followed by concurrent read-only access during runtime, avoiding contention in the hot path while maintaining correctness.

Duplicate Registration Handling#

Register() panics on duplicate registration to catch configuration conflicts at startup (following the database/sql and image.RegisterFormat pattern):

func (r *DeviceParserRegistry) Register(deviceType string, fn ConstructorFunc) {
    if fn == nil {
        panic(fmt.Sprintf("parser: cannot register nil factory for device type %q", deviceType))
    }

    key := strings.ToLower(strings.TrimSpace(deviceType))
    if key == "" {
        panic("parser: device type name cannot be empty")
    }

    r.mu.Lock()
    defer r.mu.Unlock()

    if _, exists := r.parsers[key]; exists {
        panic(fmt.Sprintf("parser: duplicate registration for device type %q", key))
    }

    r.parsers[key] = fn
}

This fail-fast approach detects plugin conflicts during application startup rather than silently overwriting parsers or exhibiting undefined behavior at runtime.

Auto-Detection with Registry#

The auto-detection logic uses registry lookup to resolve device types:

func (f *Factory) createWithAutoDetect(ctx context.Context, r io.Reader, validateMode bool) (*common.CommonDevice, error) {
    rootElem, fullReader, err := peekRootElementBounded(ctx, r)
    if err != nil {
        return nil, err
    }

    fn, ok := f.registry.Get(rootElem)
    if !ok {
        return nil, fmt.Errorf(
            "unsupported device type: root element <%s> is not recognized; supported: %s",
            rootElem, strings.Join(f.registry.List(), ", "),
        )
    }

    return parseDevice(ctx, fn(f.xmlDecoder), fullReader, validateMode)
}

This implementation maintains the existing auto-detection behavior while delegating parser selection to the registry, enabling extensibility without core logic changes.

Built-in Parser Registration#

OPNsense parser self-registers at package initialization using Go's init() mechanism:

// pkg/parser/opnsense/parser.go
func init() {
    parser.Register("opnsense", NewParserFactory)
}

// NewParserFactory returns a new DeviceParser configured for OPNsense devices.
// It satisfies the factory function signature required by DeviceParserRegistry.
func NewParserFactory(decoder parser.OPNsenseXMLDecoder) parser.DeviceParser {
    return NewParser(decoder)
}

This pattern ensures built-in parsers are automatically registered when the package is imported, requiring no manual registration code in application entry points. The parser.Register() function is a package-level convenience wrapper around DefaultRegistry().Register().

External Plugin Pattern (Compile-Time Linking)#

v2.0.0 supports compile-time linking for external parsers via blank imports:

// External repo: github.com/example/pfsense-parser
package pfsense

import "github.com/EvilBit-Labs/opnDossier/pkg/parser"

func init() {
    parser.Register("pfsense", NewParserFactory)
}

func NewParserFactory(decoder parser.OPNsenseXMLDecoder) parser.DeviceParser {
    return &PfSenseParser{decoder: decoder}
}

Consumer applications link the plugin via blank import, triggering registration at initialization:

// cmd/custom-opndossier/main.go
package main

import (
    "github.com/EvilBit-Labs/opnDossier/cmd"
    _ "github.com/example/pfsense-parser" // registers at init()
)

func main() { cmd.Execute() }

Tradeoffs:

• ✅ Simple, type-safe, zero IPC overhead
• ✅ Standard Go pattern (used by database/sql, image, buf)
• ❌ Requires rebuilding opnDossier binary with plugin import

Future (v2.1.0+): Subprocess plugins using HashiCorp go-plugin style for dynamic plugin discovery without recompilation (out of scope for v2.0.0).

Comparison with Existing Plugin System#

The parser registry follows the same pattern as opnDossier's compliance plugin registry:

// internal/audit/plugin.go
type PluginRegistry struct {
    plugins map[string]compliance.Plugin
    mutex sync.RWMutex
}

func (pr *PluginRegistry) RegisterPlugin(p compliance.Plugin) error {
    pr.mutex.Lock()
    defer pr.mutex.Unlock()

    if _, exists := pr.plugins[p.Name()]; exists {
        return fmt.Errorf("plugin %s is already registered", p.Name())
    }

    pr.plugins[p.Name()] = p
    return nil
}

This consistency reduces cognitive load for contributors already familiar with the compliance plugin system, providing a unified mental model for extensibility across opnDossier's architecture.

Acceptance Criteria#

The v2.0.0 parser registry implementation is complete:

• Thread-safe DeviceParserRegistry with Register, Get, List methods
• Factory.createWithOverride uses registry.Get with (fn, ok) lookup
• Factory.createWithAutoDetect uses registry.Get for root element matching
• OPNsense parser self-registers via init() calling parser.Register("opnsense", NewParserFactory)
• Error messages dynamically list supported types from registry.List()
• cmd/shared_flags.go ValidDeviceTypes and validateDeviceType use registry
• Comprehensive test coverage in pkg/parser/registry_test.go (9 test functions)

The implementation is documented in docs/solutions/architecture-issues/pluggable-deviceparser-registry-pattern.md.

Re-Export Layer Elimination (COMPLETED)#

Removal in PR #404#

PR #404 eliminated the 92 type aliases in internal/model/ originally planned for removal in v2.0.0. The entire internal/model/ directory was deleted:

This deletion removed 1,303 lines of alias code that provided no functional value beyond backward compatibility, significantly simplifying the codebase.

Benefits of Elimination#

The re-export layer elimination provides several concrete benefits:

1. Reduced cognitive load — Single source of truth for type definitions eliminates mental mapping between alias locations and actual implementations
2. Improved IDE navigation — Go-to-definition navigates directly to implementation rather than intermediate alias layer
3. Simplified maintenance — No synchronization required between alias layer and source types; updates propagate directly
4. Clearer API boundaries — Public packages explicitly defined in pkg/, making external consumption patterns obvious
5. Removed duplication — No need to track which types are re-exported or maintain parallel import paths

These improvements reduce the barrier to entry for new contributors and eliminate an entire class of maintenance tasks.

Lessons Learned: Package Boundary Violations#

The restructuring revealed that mechanical import path updates are insufficient when exposing internal/ packages as public APIs. Four files initially violated Go's internal/ access boundary:

• pkg/parser/factory.go — imported internal/cfgparser for DefaultMaxInputSize
• pkg/parser/opnsense/parser.go — imported internal/cfgparser for NewXMLParser()
• pkg/schema/opnsense/common.go — imported internal/constants for NetworkAny
• pkg/schema/opnsense/security.go — imported internal/constants for NetworkAny

Resolution patterns:

1. Constant extraction — For simple constants like NetworkAny, define local copies in pkg/ to eliminate dependency chains
2. Interface injection — For complex dependencies like cfgparser, define interfaces in pkg/ and inject concrete implementations at the cmd/ layer

These patterns prevent cascading moves of entire dependency chains (e.g., moving cfgparser would require moving validator and constants). The interface injection pattern is now the standard approach for pkg/ packages requiring internal/ functionality.

Prevention checklist for future public package exposures:

• Run boundary verification: grep -rn 'internal/' --include='*.go' pkg/ | grep -v _test.go
• Audit struct fields for leaked internal/ types (define public equivalents in pkg/)
• Consider CI enforcement to catch boundary violations automatically
• Document interface contracts in pkg/ for external implementers

Breaking Change Communication#

The restructuring was a breaking change delivered in a single PR:

• No deprecation period — internal/model/ re-export layer immediately removed
• Migration accomplished atomically — All 173 files updated in PR #404
• External consumers — External projects must update imports to use pkg/ packages directly

Since opnDossier's public packages were previously in internal/, external consumption was not possible prior to PR #404. The breaking change only affects internal opnDossier code, which was updated atomically in the same PR.

Design Principles Maintained#

The restructuring preserves all five core design principles of opnDossier:

1. Offline-First — Zero new external dependencies; complete air-gap compatibility maintained. The restructuring introduces no network-dependent functionality and preserves the ability to operate in isolated environments.

2. Operator-Focused — No changes to CLI interface or operator workflows. All commands (convert, audit, diff, display, validate) maintain identical behavior and flag sets.

3. Framework-First — Continues leveraging established Go libraries (encoding/xml, standard library). The restructuring uses only Go's built-in sync primitives and no additional dependencies.

4. Structured Data — Configuration hierarchy and relationships unchanged. The CommonDevice model preserves all structural information from device-specific schemas.

5. Security-First — No telemetry, input validation remains strict, secure processing unchanged. The parser registry adds no attack surface beyond existing XML parsing.

The package reorganization is purely an internal architecture improvement with no impact on runtime behavior, dependency footprint, or security posture.

Related Topics#

Multi-Device Support Roadmap#

The restructuring enables support for additional device types planned for future releases:

• pfSense — Community-requested firewall platform with similar XML configuration format
• Cisco ASA — Enterprise firewall support requiring custom schema mapping
• Fortinet FortiGate — Post-v2.0.0 consideration for multi-vendor environments
• MikroTik RouterOS — Post-v2.0.0 consideration for network appliance support

Each new device type will implement the DeviceParser interface and register itself via init(), requiring no changes to core opnDossier code. This extensibility model follows the Open/Closed Principle: opnDossier is open for extension (new device types) but closed for modification (stable core).

Compliance Plugin Architecture#

The parser registry pattern mirrors the existing compliance plugin system, which provides:

• Built-in plugins: Firewall, SANS, STIG
• Thread-safe registry with sync.RWMutex
• Duplicate plugin detection
• Sequential initialization at startup

External parser plugins follow the same lifecycle and registration patterns, providing a consistent extensibility model across opnDossier's architecture. This consistency reduces learning overhead and enables contributors familiar with compliance plugins to quickly understand parser plugin development.

Two-Stage Parsing Architecture#

The package restructuring maintains the two-stage parsing design that separates XML concerns from domain logic:

1. Stage 1: XML Parsing — DeviceParser reads device-specific XML into schema DTOs (pkg/schema/opnsense/)
2. Stage 2: Domain Conversion — CommonConverter transforms schema DTOs into platform-agnostic CommonDevice (pkg/model/)

This separation enables compliance plugins to access device-specific fields via RawConfig while operating primarily on the CommonDevice abstraction. The two-stage design ensures that adding new device types requires only implementing device-specific parsers and converters, without modifying downstream consumers (audit plugins, converters, processors).

Relevant Code Files#