Native Library Build and Linking#

Overview#

RAGFlow's Go server statically links three native libraries for PDF and Office document parsing:

All three are statically linked (.a) via CGO — dynamic shared libraries (.so) were abandoned in PR #16496 to eliminate runtime LD_LIBRARY_PATH dependencies and improve binary portability.

The build lifecycle has two entry points:

• ragflow_deps/download_go_deps.py — downloads and extracts archives to ~/ragflow-native-libs/
• build.sh — checks, configures CGO_LDFLAGS, and compiles the Go server against the static archives

Library Versions and Download#

Pinned versions are declared at the top of build.sh:

Note: download_go_deps.py downloads pdf_oxide v0.3.73 while build.sh pins v0.3.67 — these need to be kept in sync manually.

download_go_deps.py extracts archives into:

• ~/ragflow-native-libs/pdfium-static/
• ~/ragflow-native-libs/pdf_oxide/
• ~/ragflow-native-libs/office_oxide/

It skips extraction if the target directory already exists 1. Pass --china-mirrors to route downloads through gh-proxy.com 2.

In CI environments, libraries are pre-seeded at /opt/ragflow-native-libs/. The _seed_from_system() function in build.sh copies from there to the user cache if needed, skipping network access entirely 3.

The Go module wrappers are pinned in go.mod:

• github.com/yfedoseev/pdf_oxide/go v0.3.67
• github.com/yfedoseev/office_oxide/go v0.1.8

CGO Static Linking#

setup_cgo_env() in build.sh builds CGO_CFLAGS and CGO_LDFLAGS before invoking go build. The flags are assembled in order: office_oxide → pdfium → pdf_oxide → system libs.

LLVM/lld Requirement (Linux only)#

pdfium is built by Chromium's toolchain using Clang, which emits .eh_frame exception-handling sections in a format GNU ld cannot merge. On Linux, lld (LLVM linker) is required 4:

-fuse-ld=lld -Wl,--allow-multiple-definition

ld.lld must be on $PATH; the script aborts with installation instructions if it is missing 5.

The libc++.a and libc++abi.a bundled inside the pdfium-static tarball are also linked on Linux 6.

macOS uses the system linker with -framework CoreGraphics/CoreFoundation/Security/SystemConfiguration 7.

Rust Symbol Conflict Resolution#

Both pdf_oxide and office_oxide are compiled as Rust staticlib crates that each embed a copy of the Rust runtime. Linking both into the same binary produces duplicate rust_eh_personality symbols. The --allow-multiple-definition flag passed to lld silences this — it is not a workaround but the correct approach for this Rust multi-staticlib scenario 8 9.

pdf_oxide Platform Subdirectory#

The pdf_oxide go-ffi tarball places the .a under a platform-specific subdirectory (linux_amd64, linux_arm64, darwin_amd64, darwin_arm64). setup_cgo_env() detects the current platform and builds the correct path at link time 10.

Version Verification and Go Build Cache Invalidation#

Runtime Version Check#

check_office_oxide_deps() uses strings on the .a archive to verify the on-disk library matches the pinned version. If the version string is absent, the build aborts with 11:

"A stale lib silently loses PPT97 (.ppt) slide content."

This guards against a silent data-loss bug (yfedoseev/office_oxide#85) where PlainText() on a legacy .ppt file returns stale metadata instead of slide text — fixed in v0.1.8, introduced in PR #17262.

Go Build Cache Invalidation via Version-Stamped Symlinks#

Go's build cache keys CGO_LDFLAGS as a string — it does not hash the .a file content. Swapping the archive in-place at the same path is invisible to the cache, causing go build to silently reuse a stale binary.

setup_cgo_env() solves this by creating a version-stamped symlink 12:

~/ragflow-native-libs/office_oxide/lib/v0.1.8/liboffice_oxide.a → ../liboffice_oxide.a

The symlink directory path is embedded in CGO_LDFLAGS. When the version changes (e.g., 0.1.8 → 0.1.9), the flag string changes, forcing a full relink 13.

PDFium Thread Safety#

PDFium is documented as not thread-safe. Concurrent FPDF_* calls across goroutines corrupt the heap and cause SIGSEGV. This is especially problematic because both the CGO pdfium binding and the Rust pdf_oxide binding link the same PDFium static instance — two separate per-binding mutexes would not prevent races between them.

PR #16795 introduced internal/deepdoc/parser/pdf/pdfsync/pdfsync.go, a process-wide shared mutex (pdfsync.Mu) that serializes all PDFium calls across both bindings. All PDF entry points (PageSize, RenderPage, ExtractOutlines, GetPageChars, etc.) are wrapped with pdfsync.With(f) or pdfsync.WithErr(f) 14.

This serialization is acceptable in practice: PDF parsing is I/O-bound and the throughput loss from serialization is negligible compared to the cost of document decoding.

Historical Evolution#

The progression — dynamic → static → version-verified static — reflects iterative hardening of the native library pipeline against portability failures, silent version mismatches, and runtime crashes.

Key Source References#