Schema Parser Synchronization#

Lead Section#

Schema Parser Synchronization is a critical architectural pattern in the opnDossier project that maintains consistency between XML schema definitions, parser logic, and data converters. The system transforms OPNsense firewall configuration XML files into platform-agnostic Go data models through a three-layer architecture where field names, types, and structures must remain synchronized to ensure data integrity.

The pattern encompasses two fundamental coordination mechanisms: the Bridge/VLAN container pattern for parent-child XML structures, and the temp-variable-append pattern for transforming XML repeating elements into Go slices. These patterns enable proper marshaling of complex nested configurations while maintaining type safety and performance.

Schema Parser Synchronization is essential for maintaining architectural separation where the XML DTO layer remains untouched by downstream consumers, ensuring that changes to field names, singular-to-slice transformations, or nested-to-flat restructuring are properly coordinated across all three layers to prevent data loss.

Architecture Overview#

Three-Layer Architecture#

The opnDossier parser implements a strict three-layer separation:

Layer 1: XML DTO Layer (pkg/schema/opnsense/)

Carries xml:"" tags and mirrors OPNsense config.xml structure exactly
Untouched by downstream consumers
Defines struct fields with XMLName as first field for proper marshaling

Layer 2: XML Parser (internal/cfgparser/xml.go)

Uses Go's built-in encoding/xml package to deserialize XML
Implements switch-based element dispatching referencing OpnSenseDocument fields by XML tag name
Returns schema.OpnSenseDocument without transformation

Layer 3: Data Converter (pkg/parser/opnsense/)

Only package that imports pkg/schema/opnsense/
Converts schema.OpnSenseDocument to *common.CommonDevice
Applies field mappings, restructuring, and type conversions

Synchronization Points#

Synchronization requirements exist at three critical junctions:

XML Tag to Schema Field: XML element names in config.xml must match struct tag definitions
Schema Field to Parser Switch: Switch case names must match XML tags (not Go field names)
Schema Field to Converter Logic: Converters must reference correct field paths and handle type transformations

Bridge/VLAN Container Pattern#

The Bridge/VLAN pattern is a consistent container/child structure for XML marshaling where parent structs hold child collections.

Pattern Structure#

Container (Parent) Struct:

type VLANs struct {
    XMLName xml.Name `xml:"vlans"` // Must be first field
    VLAN []VLAN `xml:"vlan,omitempty"` // Slice of children
}

Child Struct:

type VLAN struct {
    XMLName xml.Name `xml:"vlan"` // Must be first field
    If string `xml:"if,omitempty"`
    Tag string `xml:"tag,omitempty"`
    Descr string `xml:"descr,omitempty"`
}

Key Characteristics#

XMLName as first field in both container and child structs
Container uses []ChildType pattern for slice fields
Plural container names (VLANs, Bridges, Gateways) vs singular child names (VLAN, Bridge, Gateway)
XML tags match convention: plural container tag, singular child tag

Examples from Codebase#

Bridges/Bridge Pattern:

type Bridges struct {
    XMLName xml.Name `xml:"bridges"`
    Bridge []Bridge `xml:"bridge,omitempty"`
}

type Bridge struct {
    XMLName xml.Name `xml:"bridge"`
    Members string `xml:"members,omitempty"`
    Descr string `xml:"descr,omitempty"`
    Bridgeif string `xml:"bridgeif,omitempty"`
    STP BoolFlag `xml:"stp,omitempty"`
}

Gateways/Gateway Pattern:

type Gateways struct {
    XMLName xml.Name `xml:"gateways"`
    Gateway []Gateway `xml:"gateway_item,omitempty"`
    Groups []GatewayGroup `xml:"gateway_group,omitempty"`
}

type Gateway struct {
    XMLName xml.Name `xml:"gateway_item"`
    Interface string `xml:"interface,omitempty"`
    Gateway string `xml:"gateway,omitempty"`
    Name string `xml:"name,omitempty"`
    Weight string `xml:"weight,omitempty"`
}

Additional Container Patterns:

Temp-Variable-Append Pattern#

The temp-variable-append pattern is used consistently across converter functions to transform XML repeating elements into Go slices.

Pattern Steps#

Check for empty source → Return nil
Pre-allocate temporary slice with capacity hint
Loop through source elements
Append transformed elements using inline struct literals
Return temporary slice

Example: Converting Bridges#

From converter_network.go:

func (c *Converter) convertBridges(doc *schema.OpnSenseDocument) []common.Bridge {
    if len(doc.Bridges.Bridge) == 0 {
        return nil // Step 1: Return nil for empty
    }

    // Step 2: Pre-allocate with capacity
    result := make([]common.Bridge, 0, len(doc.Bridges.Bridge))

    // Step 3: Loop through source
    for _, b := range doc.Bridges.Bridge {
        // Step 4: Append transformed element
        result = append(result, common.Bridge{
            BridgeIf: b.Bridgeif,
            Members: splitNonEmpty(b.Members, ","),
            Description: b.Descr,
            STP: bool(b.STP),
            Created: b.Created,
            Updated: b.Updated,
        })
    }

    // Step 5: Return result
    return result
}

Additional Examples#

Converting VLANs:

func (c *Converter) convertVLANs(doc *schema.OpnSenseDocument) []common.VLAN {
    if len(doc.VLANs.VLAN) == 0 {
        return nil
    }

    result := make([]common.VLAN, 0, len(doc.VLANs.VLAN))
    for _, v := range doc.VLANs.VLAN {
        result = append(result, common.VLAN{
            PhysicalIf: v.If,
            Tag: v.Tag,
            Description: v.Descr,
            VLANIf: v.Vlanif,
        })
    }

    return result
}

Converting Certificates:

func (c *Converter) convertCertificates(doc *schema.OpnSenseDocument) []common.Certificate {
    if len(doc.Certs) == 0 {
        return nil
    }

    result := make([]common.Certificate, 0, len(doc.Certs))
    for _, cert := range doc.Certs {
        result = append(result, common.Certificate{
            RefID: cert.Refid,
            Description: cert.Descr,
            Certificate: cert.Crt,
            PrivateKey: cert.Prv,
        })
    }

    return result
}

Nested Pattern: Users with API Keys:

func (c *Converter) convertUsers(doc *schema.OpnSenseDocument) []common.User {
    if len(doc.System.User) == 0 {
        return nil
    }

    result := make([]common.User, 0, len(doc.System.User))
    for _, u := range doc.System.User {
        user := common.User{
            Name: u.Name,
            Description: u.Descr,
        }

        // Nested application of pattern
        if len(u.APIKeys) > 0 {
            user.APIKeys = make([]common.APIKey, 0, len(u.APIKeys))
            for _, k := range u.APIKeys {
                user.APIKeys = append(user.APIKeys, common.APIKey{
                    Key: k.Key,
                    Secret: k.Secret,
                    Privileges: k.Privileges,
                })
            }
        }

        result = append(result, user)
    }

    return result
}

Performance Optimization#

The pattern uses pre-allocated slices with capacity hints:

result := make([]Type, 0, len(sourceSlice))

This avoids repeated memory reallocations during append operations, improving performance for large configuration files.

Critical Synchronization Gotchas#

Gotcha 1: Parser Switch Cases Must Use XML Tag Names#

The parser's switch statement uses XML tag names (from struct tags), not Go field names.

Correct field name reference:

case "nat": // XML tag name (correct)
    return decodeSection(dec, &doc.Nat, se) // Go field name

XML-to-Go naming mappings:

case "hasync" → doc.HighAvailabilitySync
case "ifgroups" → doc.InterfaceGroups
case "staticroutes" → doc.StaticRoutes

From OpnSenseDocument schema:

type OpnSenseDocument struct {
    Nat Nat `xml:"nat,omitempty"` // XML: nat, Go: Nat
    Filter Filter `xml:"filter,omitempty"` // XML: filter, Go: Filter
    Interfaces Interfaces `xml:"interfaces,omitempty"` // XML: interfaces, Go: Interfaces
}

Gotcha 2: XMLName Must Be First Field#

XMLName must be first field for proper XML marshaling:

Incorrect:

type Rule struct {
    Protocol string
    XMLName xml.Name `xml:"rule"` // Second position breaks marshaling
}

Correct:

type Rule struct {
    XMLName xml.Name `xml:"rule"` // First position
    Protocol string `xml:"protocol"`
}

Gotcha 3: Manual Append for Top-Level Repeating Elements#

When XML elements repeat at document root (e.g., <ca>, <cert>), the parser manually appends to slices:

case "ca":
    var ca schema.CertificateAuthority
    if err := decodeSection(dec, &ca, se); err != nil {
        return err
    }
    doc.CAs = append(doc.CAs, ca) // Manual append required
    return nil

This is necessary because Go's encoding/xml requires either multiple sibling elements or a container element to populate slices automatically.

Gotcha 4: Nested vs Top-Level Field Distinction#

Source.Port vs Rule.SourcePort serve different purposes:

Nested Port (matching):

type Source struct {
    Port string `xml:"port,omitempty"` // Inside <source><port>...</port></source>
}

Top-Level Port (translation):

type NATRule struct {
    SourcePort string `xml:"sourceport,omitempty"` // Top-level <sourceport>...</sourceport>
}

Converter must extract from both:

Source: common.RuleEndpoint{
    Port: rule.Source.Port, // Nested port from <source>
},
SourcePort: rule.SourcePort, // Top-level translated port

Gotcha 5: Field Priority Logic#

The EffectiveAddress() method implements strict priority: Network > Address > Any

func (s Source) EffectiveAddress() string {
    if s.Network != "" {
        return s.Network // Highest priority
    }
    if s.Address != "" {
        return s.Address // Second priority
    }
    if s.IsAny() {
        return constants.NetworkAny // Third priority
    }
    return ""
}

Fields are mutually exclusive in OPNsense semantics—only one should be present per source/destination block.

Gotcha 6: Singular to Slice Transformations#

The InterfaceList type automatically transforms comma-separated XML into slices:

XML Input:

<interface>wan,lan,opt1</interface>

Custom UnmarshalXML:

type InterfaceList []string

func (il *InterfaceList) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
    var content string
    if err := d.DecodeElement(&content, &start); err != nil {
        return err
    }

    parts := strings.Split(content, ",")
    interfaces := make([]string, 0, len(parts))
    for _, part := range parts {
        if trimmed := strings.TrimSpace(part); trimmed != "" {
            interfaces = append(interfaces, trimmed)
        }
    }

    *il = InterfaceList(interfaces)
    return nil
}

Result: []string{"wan", "lan", "opt1"}

Other comma-separated fields use splitNonEmpty() helper:

Members: splitNonEmpty(b.Members, ",") // Bridge members

Exception: Interface groups use space separators:

Members: splitNonEmpty(e.Members, " ") // Note space, not comma

Gotcha 7: Package Name Extraction#

The convertPackages function extracts firmware plugin names from comma-separated strings:

func (c *Converter) convertPackages(doc *schema.OpnSenseDocument) []common.Package {
    names := splitNonEmpty(doc.System.Firmware.Plugins, ",")
    if len(names) == 0 {
        return nil
    }

    result := make([]common.Package, 0, len(names))
    for _, name := range names {
        result = append(result, common.Package{
            Name: name,
            Type: packageTypePlugin,
            Installed: true, // All in config.xml are assumed installed
        })
    }

    return result
}

XML Input:

<firmware>
  <plugins>os-firewall,os-acb,os-theme-cicada</plugins>
</firmware>

Output:

[]common.Package{
    {Name: "os-firewall", Type: "plugin", Installed: true},
    {Name: "os-acb", Type: "plugin", Installed: true},
    {Name: "os-theme-cicada", Type: "plugin", Installed: true},
}

The converter creates Package structs with minimal metadata. Full package details (versions, descriptions) require the OPNsense API.

Best Practices#

1. Single Source of Truth#

The schema package is authoritative for XML structure. Never replicate field definitions. Only pkg/parser/opnsense/ imports pkg/schema/opnsense/.

2. Document Field Transformations#

Every converter function should document mappings:

// convertRule transforms schema.Rule to CommonRule
// Field mappings:
// - rule.Source.Port → CommonRule.SourcePort (extracted from nested)
// - rule.Source (network/address/any) → CommonRule.SourceAddress (priority: network > address > any)
func convertRule(xmlRule schema.Rule) CommonRule {
    // Implementation
}

3. Enforce Bridge Pattern Consistently#

XMLName first field in containers and children
Container uses []ChildType pattern
Use linting or review checklist

4. Test-Driven Verification#

When adding new schema fields:

Update schema struct with correct XML tag
Run parser against sample config.xml
Verify parser correctly unmarshals field
Implement converter logic
Test roundtrip: XML → schema → CommonDevice → output

5. Handle Empty Inputs Consistently#

Return nil for empty slices (not []Type{}):

if len(doc.Bridges.Bridge) == 0 {
    return nil // Consistent empty handling
}

6. Pre-Allocate Slices#

Use capacity hints for performance:

result := make([]common.Bridge, 0, len(doc.Bridges.Bridge))

7. Maintain Layer Separation#

Schema layer: XML DTO only, no business logic
Parser layer: Deserialization only, no transformation
Converter layer: All transformation logic, no XML details

Relevant Code Files#

File Path	Description	Lines of Interest
`pkg/schema/opnsense/opnsense.go`	Root OpnSenseDocument definition with XML tags	9-46
`pkg/schema/opnsense/interfaces.go`	Bridge/VLAN pattern examples (VLANs, Bridges, GIF, GRE, LAGG)	208-240
`pkg/schema/opnsense/network.go`	Gateways/Gateway, StaticRoutes/StaticRoute patterns	29-79
`pkg/schema/opnsense/security.go`	InterfaceList, Source/Destination with EffectiveAddress()	12-274
`internal/cfgparser/xml.go`	Parser switch cases, decodeSection, manual append for CA/Cert	132-252
`pkg/parser/opnsense/converter.go`	Main converter entry point, firewall rules conversion	168-278
`pkg/parser/opnsense/converter_network.go`	Temp-variable-append pattern examples (Bridges, VIPs, Interface Groups)	11-154
`pkg/parser/opnsense/converter_security.go`	Certificates, CAs, and Packages conversion with append pattern	11-73
`pkg/parser/opnsense/converter_services.go`	Nested pattern (Users/APIKeys), splitNonEmpty() helper	96-701

opnDossier Architecture - Three-layer architecture and data flow
XML Structure Research - OPNsense XML field semantics and standards
CommonDevice Model - Platform-agnostic domain model
Go encoding/xml Package - Standard library XML marshaling/unmarshaling
Firewall Configuration Management - Cross-platform configuration parsing patterns