OpenTelemetry Global Tracer Provider#

Overview#

The OpenTelemetry (OTEL) global tracer provider is a process-wide singleton used to resolve tracers anywhere in a Node.js application. In Phoenix's JavaScript stack, this singleton is a key source of non-deterministic telemetry capture in test environments: when multiple suites or experiment phases share a process (as Vitest workers do), one suite installing a real exporting provider as the global can silently make other suites—which believe they are running in no-op mode—emit live telemetry.

Core Problem: Global State Contamination#

Two concrete failure modes exist, addressed by separate fixes:

1. Vitest Suite Cross-Contamination (PHOENIX_TEST_TRACKING=false Unreliable)#

Issue: #13930 – PHOENIX_TEST_TRACKING=false does not reliably disable recording across all suites in a single Vitest run.

Root causes (diagnosed in #13930):

1. attachGlobalTracerProvider leaks across suites. In initializeSuite, a tracking-enabled suite calls register({ global: false }) followed by attachGlobalTracerProvider(provider), which mounts the real exporting provider as the process-wide global. Any code path that resolves a tracer from the global OTEL API (rather than the suite-local suite.tracer) then exports spans—even in suites whose own isTrackingEnabled gate returned false.

2. Module-level allSuites shared state. runner.ts declares a module-level const allSuites: SuiteState[] = [] that all suites push into and that persists for the lifetime of the worker process. The clearAllSuites() reset must be called by the reporter at the start of each run; if it isn't called (e.g. in watch mode), stale state from a prior run remains.

3. warnedAboutHttpScheme module-level flag is another example of shared state that persists across suites, confirming the broader pattern.

Fix (landed on main): phoenix-test-tracking.ts now captures PHOENIX_TEST_TRACKING at module load time into trackingDisabledAtLoad and a sticky trackingLatchedOff boolean. Once tracking is seen disabled—either at load time or on any isTrackingEnabled call—trackingLatchedOff latches to true for the entire process lifetime. This prevents a later-initializing tracking-enabled suite from re-opening the global provider. The isTrackingEnabled function checks the latch before any env var read.

2. Experiment Phase Collisions (runExperiment Task/Eval Phases)#

Issue: PR #12303 – When runExperiment ran task then evaluator phases, each phase called provider.register() to set itself as the global OTEL tracer provider. The second call silently replaced the first, orphaning task-phase spans and blocking clean shutdown.

Fix: PR #12303 introduced attachGlobalTracerProvider / detachGlobalTracerProvider with stack-based semantics into phoenix-otel/src/register.ts.

Key API: attachGlobalTracerProvider / detachGlobalTracerProvider#

Defined in js/packages/phoenix-otel/src/register.ts.

• attachGlobalTracerProvider(provider): Takes a snapshot of the current global OTEL state before mounting the new provider. Pushes the provider onto a managedGlobalTracerProviderMounts stack. Returns a GlobalTracerProviderRegistration handle with a detach() method.
• detachGlobalTracerProvider(): Pops the top mount from the stack. When the stack is empty, restores the pre-experiment snapshot. Out-of-order detaches are silently ignored.
• setGlobalProvider(provider): Routes all global-state mutations through phoenix-otel's own imports of trace/context/propagation to avoid pnpm workspace module-identity issues where the SDK's internal @opentelemetry/api import resolves to a different singleton.

When register({ global: true }) is called, it internally calls attachGlobalTracerProvider and binds a detach to provider.shutdown() so cleanup is automatic.

pnpm Workspace Module Identity Issue#

In pnpm workspaces, NodeTracerProvider.register() from the OTel SDK calls trace.setGlobalTracerProvider() using the SDK's own internal import of @opentelemetry/api. If that import resolves to a different module instance than the one in phoenix-otel (different symlink paths → different TraceAPI singletons), snapshot/restore operates on the wrong proxy and the user's original global provider is never properly restored. setGlobalProvider fixes this by always going through phoenix-otel's own imported symbols.

Key Files#

Related Issues / PRs#

• Issue #13930: PHOENIX_TEST_TRACKING=false non-determinism in Vitest suites — root cause analysis and fix description
• PR #12303: Initial attachGlobalTracerProvider/detachGlobalTracerProvider introduction, experiment phase collision fix