XML Presence Detection#

XML Presence Detection is a critical pattern in the opnDossier project for correctly parsing and generating OPNsense firewall configuration files. The pattern addresses a fundamental challenge in Go's encoding/xml package: distinguishing between XML elements that are present (self-closing tags like <disabled/>) versus completely absent elements, which both unmarshal to empty strings when using plain string fields.

The opnDossier project implements two complementary solutions: the BoolFlag custom type for presence-based boolean semantics, and pointer strings (*string) for fields where element presence must be distinguished from absence. These patterns map directly to OPNsense's PHP codebase, which uses isset() to check element presence rather than testing string values.

Understanding XML presence detection is essential for correctly implementing OPNsense configuration parsers, as choosing the wrong Go type silently breaks semantics and can cause firewall rules to behave incorrectly.

The Problem: Go's encoding/xml Ambiguity#

Self-Closing vs Absent Elements#

Go's encoding/xml package treats both self-closing tags (<tag/>) and absent elements identically when unmarshaling into a plain string field — both produce an empty string "". This creates a semantic problem when the XML schema uses element presence to convey meaning.

For example, in OPNsense XML:

• <rule><disabled/></rule> → rule is disabled (element present)
• <rule></rule> → rule is enabled (element absent)

Using a plain string field in Go cannot distinguish these cases:

// INCORRECT: Cannot distinguish presence from absence
type Rule struct {
    Disabled string `xml:"disabled,omitempty"`
}

Both XML inputs unmarshal to Disabled: "", losing the semantic distinction.

OPNsense PHP Pattern: isset()#

OPNsense's PHP code uses isset() to check for element presence, not value equality:

if (isset($rule['disabled'])) {
    // Rule is disabled
}

This PHP pattern requires Go types that preserve presence/absence semantics.

BoolFlag Type for Presence-Based Booleans#

Definition and Implementation#

The BoolFlag type is a simple boolean type alias with custom XML marshaling:

type BoolFlag bool

The UnmarshalXML method implements presence-based semantics with value parsing:

func (bf *BoolFlag) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
    var content string
    if err := d.DecodeElement(&content, &start); err != nil {
        return err
    }

    if strings.TrimSpace(content) == "" {
        *bf = true // Empty body = presence means enabled
        return nil
    }

    *bf = BoolFlag(shared.IsValueTrue(content)) // Delegate to liberal value parser
    return nil
}

Key behavior: Empty elements (<disabled/> or <disabled></disabled>) are treated as true (presence means enabled). Elements with content delegate to shared.IsValueTrue, which recognizes "1", "on", "yes", "true", "enabled" (case-insensitive) as true and "0", "off", "no", "false", "disabled" as false. This fixes the previous behavior where <disabled>0</disabled> incorrectly parsed as true. Element absence represents false.

The MarshalXML method handles serialization:

func (bf *BoolFlag) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
    if *bf {
        return e.EncodeElement("", start) // Empty element for true
    }

    return nil // Omit element for false
}

Helper Methods#

The BoolFlag type includes three convenience methods:

// String returns string representation
func (bf *BoolFlag) String() string {
    if *bf {
        return "true"
    }
    return "false"
}

// Bool returns the underlying boolean value
func (bf *BoolFlag) Bool() bool {
    return bool(*bf)
}

// Set updates the boolean value
func (bf *BoolFlag) Set(value bool) {
    *bf = BoolFlag(value)
}

When to Use BoolFlag#

Use BoolFlag for presence-based boolean fields where:

• Element existing = true
• Element absent = false
• Element content is irrelevant
• PHP pattern: isset($rule['field']) or !empty($rule['field'])

Examples of presence-based fields that use BoolFlag:

• <disabled/> — rule is disabled
• <log/> — enable logging for this rule
• <quick/> — exit on first match
• <not/> — negate address/port match
• <any/> — match any address (in some contexts)
• <nopfsync/> — disable pfSync state synchronization
• <allowopts/> — allow packets with IP options
• <disablereplyto/> — disable reply-to for asymmetric routing
• <staticnatport/> — preserve source port in NAT

Converted BoolFlag fields in security.go:

• Rule: Disabled, Quick, Log
• NATRule: Disabled, Log
• InboundRule: Disabled, Log
• Rule (advanced fields): AllowOpts, DisableReplyTo, NoPfSync, NoSync

OPNsense Boolean Conventions#

Presence-Based vs Value-Based Booleans#

OPNsense/pfSense uses two distinct boolean patterns that require different Go type mappings:

1. Presence-Based (Use BoolFlag)#

Element existing = true, absent = false. Content is irrelevant.

• PHP pattern: isset($rule['disabled']) or !empty($rule['disabled'])
• Go type: BoolFlag
• Examples: <disabled/>, <log/>, <quick/>, <not/>

2. Value-Based (Use string)#

Element contains specific value (typically "1"). Absent or empty = false.

• PHP pattern: $config['system']['enable'] == "1"
• Go type: string
• Examples: <enable>1</enable>, <blockpriv>1</blockpriv>, <ipv6allow>1</ipv6allow>

Critical Warning: BoolFlag Misuse#

Using BoolFlag for value-based fields breaks semantics because earlier versions of BoolFlag.UnmarshalXML treated any present element as true, regardless of content. The current implementation now correctly handles value-based content by delegating non-empty bodies to shared.IsValueTrue, so <enabled>0</enabled> correctly becomes false (not true).

For fields where element presence is not semantically significant, use shared.FlexBool instead, which provides pure value-based boolean semantics without the presence layer. Both types share the same liberal vocabulary ("1", "on", "yes", "true", "enabled" for true; "0", "off", "no", "false", "disabled" for false).

Correct mapping:

// Value-based: check if value equals "1"
type System struct {
    Enable shared.FlexBool `xml:"enable,omitempty"` // CORRECT: use FlexBool for value-only semantics
}

// Presence-based: check if element exists
type Rule struct {
    Disabled BoolFlag `xml:"disabled,omitempty"` // CORRECT: use BoolFlag for presence + value
}

Pointer Strings for Presence Detection#

The *string Pattern#

The *string (pointer to string) pattern correctly distinguishes presence from absence:

• Element present (<any/> or <any></any>): unmarshals to non-nil *string pointing to ""
• Element with value (<any>value</any>): unmarshals to non-nil *string pointing to "value"
• Element absent: unmarshals to nil

Source and Destination Structs#

The Source struct uses *string for the Any field:

type Source struct {
    Any *string `xml:"any,omitempty" json:"any,omitempty" yaml:"any,omitempty"`
    Network string `xml:"network,omitempty" json:"network,omitempty" yaml:"network,omitempty"`
    Address string `xml:"address,omitempty" json:"address,omitempty" yaml:"address,omitempty"`
    Port string `xml:"port,omitempty" json:"port,omitempty" yaml:"port,omitempty"`
    Not BoolFlag `xml:"not,omitempty" json:"not,omitempty" yaml:"not,omitempty"`
}

The Destination struct has identical structure.

Code comment explaining the design:

"Any is a pointer to distinguish XML element presence (<any/> → non-nil "") from absence (nil), since Go's encoding/xml produces "" for both self-closing tags and absent elements when using a plain string."

This matches OPNsense's PHP pattern isset($this->rule['source']['any']).

Helper Methods for Safe Comparison#

IsAny() Method#

The IsAny() method for Source:

func (s Source) IsAny() bool {
    return s.Any != nil
}

The IsAny() method for Destination is identical.

Returns true when the <any> element is present in the XML (Any field is non-nil). The element's value is irrelevant — only presence matters.

EffectiveAddress() Method#

The EffectiveAddress() method for Source:

func (s Source) EffectiveAddress() string {
    if s.Network != "" {
        return s.Network
    }
    if s.Address != "" {
        return s.Address
    }
    if s.IsAny() {
        return constants.NetworkAny // "any"
    }
    return ""
}

The Destination version is identical.

This method implements OPNsense's three-tier address resolution priority (from legacyMoveAddressFields):

1. Network (highest priority) — interface names (lan, wan), interface IPs (lanip), special values like (self), or interface groups
2. Address — IP/CIDR notation or alias names (e.g., 192.168.1.0/24, MyAlias)
3. Any — if the Any element is present, returns "any"
4. Empty string if none present

The fields are mutually exclusive per OPNsense semantics: <any>, <network>, and <address> cannot coexist.

Equal() Method#

The Equal() method for Source:

func (s Source) Equal(other Source) bool {
    if (s.Any != nil) != (other.Any != nil) {
        return false
    }
    return s.Network == other.Network &&
        s.Address == other.Address &&
        s.Port == other.Port &&
        s.Not == other.Not
}

The Destination version is identical.

The Any field is compared by presence only (nil vs non-nil), not by value, because OPNsense treats <any> as a presence-based flag. This method is used for rule comparison and deduplication.

Testing XML Presence Detection#

Round-Trip Testing Pattern#

The opnDossier codebase uses comprehensive XML round-trip tests to validate presence-based semantics. These tests follow a consistent three-phase pattern that ensures both unmarshaling and marshaling preserve the correct semantics.

BoolFlag Round-Trip Tests#

Example: TestRule_BoolFlagFields_XMLRoundTrip:

func TestRule_BoolFlagFields_XMLRoundTrip(t *testing.T) {
    tests := []struct {
        name string
        xml string
        wantDisabled BoolFlag
        wantQuick BoolFlag
        wantLog BoolFlag
    }{
        {
            name: "all presence flags set",
            xml: `<rule><disabled/><quick/><log/></rule>`,
            wantDisabled: true,
            wantQuick: true,
            wantLog: true,
        },
        {
            name: "no presence flags",
            xml: `<rule></rule>`,
            wantDisabled: false,
            wantQuick: false,
            wantLog: false,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // Unmarshal from input XML
            var got Rule
            if err := xml.Unmarshal([]byte(tt.xml), &got); err != nil {
                t.Fatalf("xml.Unmarshal() error = %v", err)
            }

            // Verify fields
            if got.Disabled != tt.wantDisabled {
                t.Errorf("Disabled = %v, want %v", got.Disabled, tt.wantDisabled)
            }
            // ... (verify other fields)
        })
    }
}

Pointer String Round-Trip Tests#

Example: TestSource_XMLRoundTrip:

func TestSource_XMLRoundTrip(t *testing.T) {
    tests := []struct {
        name string
        xml string
        want Source
        wantElements []string // substrings expected in marshaled XML
    }{
        {
            name: "any self-closing",
            xml: `<source><any/></source>`,
            want: Source{Any: new("")},
            wantElements: []string{"<any>"},
        },
        {
            name: "network only",
            xml: `<source><network>lan</network></source>`,
            want: Source{Network: "lan"},
            wantElements: []string{"<network>lan</network>"},
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // Phase 1: Unmarshal from input XML
            var got Source
            if err := xml.Unmarshal([]byte(tt.xml), &got); err != nil {
                t.Fatalf("xml.Unmarshal() error = %v", err)
            }
            if !got.Equal(tt.want) {
                t.Errorf("xml.Unmarshal() = %+v, want %+v", got, tt.want)
            }

            // Phase 2: Marshal the expected struct
            marshaled, err := xml.Marshal(tt.want)
            if err != nil {
                t.Fatalf("xml.Marshal() error = %v", err)
            }

            // Verify expected elements in marshaled XML
            marshaledStr := string(marshaled)
            for _, elem := range tt.wantElements {
                if !strings.Contains(marshaledStr, elem) {
                    t.Errorf("marshaled XML missing %q", elem)
                }
            }

            // Phase 3: Unmarshal marshaled XML and compare again
            var roundTripped Source
            if err := xml.Unmarshal(marshaled, &roundTripped); err != nil {
                t.Fatalf("round-trip xml.Unmarshal() error = %v", err)
            }
            if !roundTripped.Equal(tt.want) {
                t.Errorf("round-trip = %+v, want %+v", roundTripped, tt.want)
            }
        })
    }
}

Test Pattern Summary#

All presence detection tests follow these common principles:

1. Table-driven with t.Run() subtests for clear test organization
2. Three-phase validation: unmarshal → verify → marshal → unmarshal → verify again
3. Explicit element checking in marshaled XML using strings.Contains()
4. Equality methods for comparing struct values that handle pointer fields correctly
5. Backward compatibility tests ensuring absent fields default to zero values

Additional round-trip test examples:

• TestNATRule_NewFields_XMLRoundTrip — tests StaticNatPort and NoNat BoolFlag fields
• TestInboundRule_NewFields_XMLRoundTrip — tests NoRDR and NoSync BoolFlag fields
• TestRule_CompleteRule_XMLRoundTrip — comprehensive test with all fields

Best Practices#

Creating Pointer String Values#

When working with pointer strings in Go, use these patterns for clarity:

// Helper function for clarity
func stringPtr(s string) *string {
    return &s
}

// Usage in struct initialization
source := Source{
    Any: stringPtr(""), // Element present, empty value
}

// Or using Go's address-of operator with a temporary variable
emptyStr := ""
source := Source{
    Any: &emptyStr, // Element present
}

// Nil for absent element
source := Source{
    Any: nil, // Element absent
}

XML Struct Tags#

Always use xml:"fieldname,omitempty" to omit empty fields during marshaling:

type Rule struct {
    Disabled BoolFlag `xml:"disabled,omitempty"`
    Log BoolFlag `xml:"log,omitempty"`
}

The omitempty tag ensures that false BoolFlag values and nil pointer fields are not serialized to XML, maintaining clean output.

Never Compare Pointers Directly#

WRONG:

if source.Any == destination.Any { // Compares pointer addresses, not values!
    // ...
}

CORRECT:

if source.IsAny() && destination.IsAny() { // Use helper methods
    // ...
}

// Or use Equal() method
if source.Equal(other) {
    // ...
}

Direct pointer comparison checks memory addresses, not semantic equivalence, which leads to incorrect results.

Choosing the Right Type#

Use this decision tree when mapping XML fields to Go types:

1. Does the element's presence (vs absence) convey meaning?

   • YES → Use BoolFlag (if boolean) or *string (if needs value)
   • NO → Use plain string

2. Does OPNsense PHP code use isset() or !empty() on this field?

   • YES → Use BoolFlag or *string
   • NO → Check if it uses == "1" or value comparison

3. Is this a boolean that tests for specific value like "1"?

   • YES → Use string (value-based)
   • NO → Use BoolFlag (presence-based)

Historical Context#

pfSense Bug #6893#

Prior to pfSense 2.3.3, some code produced <tag/> while other code produced <tag></tag>. Both forms are valid XML and Go's encoding/xml handles both correctly via the *string / BoolFlag pattern. The presence detection patterns in opnDossier ensure compatibility with both XML forms, making the implementation robust across different versions of configuration files.

Schema Evolution#

The config.xml data model is enhanced in phases to ensure backward compatibility:

• Phase 1: Resolved Source/Destination gaps (added missing Address, Not, Port fields)
• Phase 2: Converted high-priority Rule fields to correct types (BoolFlag conversions)

Adding new fields requires checking upstream OPNsense/pfSense source for field semantics (presence-based vs value-based) to ensure correct type mapping.

Development Standards#

Go Version Requirements#

The project targets Go 1.26+ as minimum supported version, with development toolchain go1.21.7 and system go1.25.7.

Code Style#

Follow Google Go Style Guide conventions:

• camelCase for private identifiers
• PascalCase for exported identifiers
• Tabs for indentation
• 80-120 character line length

Code Quality Tools#

Use automated quality enforcement:

• gofmt — standard formatting
• gofumpt — stricter formatting
• golangci-lint — comprehensive linting
• go vet — suspicious code detection
• gosec — security scanning

Testing Standards#

Testing requirements:

• Coverage target: >80%
• Test pattern: Table-driven tests using t.Run() subtests
• Performance: Individual tests <100ms
• Round-trip tests: Required for all XML presence-detection fields

Relevant Code Files#

Related Topics#

• Go XML Marshaling — Understanding Go's encoding/xml package behavior with self-closing tags and empty elements
• OPNsense Configuration Schema — Structure and semantics of OPNsense config.xml files
• pfSense PHP Codebase — Upstream PHP patterns using isset() and value-based boolean checks
• Type-Safe Configuration Parsing — Using Go's type system to enforce configuration semantics
• Firewall Rule Validation — Validating firewall rules parsed from XML with presence-based semantics

See Also#

• OPNsense Documentation
• Go encoding/xml Package
• opnDossier GitHub Repository
• pfSense XML Configuration Reference