Document Ingestion Data Integrity#

Two distinct data integrity bugs affect the documents and failed_documents ingestion pipelines:

1. Null byte sanitization is applied only at OCR pipeline exit points, leaving database insert paths unprotected. Non-OCR text (e.g., filenames, error messages) can reach PostgreSQL containing \0 characters, which PostgreSQL rejects in TEXT columns.
2. failure_reason constraint mismatch: the ingestion service writes "storage_error" and "database_error" as failure_reason values, but neither appears in the failed_documents CHECK constraint allowlist. The resulting constraint violation is silently swallowed, so no failure record is ever persisted.

Null Byte Sanitization — Where It Is (and Isn't)#

What was fixed#

PR #308 (merged 2025-11-04) 1 added remove_null_bytes() calls at every OCR pipeline exit point after issue #296 revealed that PDF and document parsers can embed \0 characters in extracted text, causing PostgreSQL TEXT column rejections. The function is defined at enhanced.rs:107–123 and is called at five points within EnhancedOcrService:

• Line 174 — after Tesseract image OCR
• Line 987 — after pdftotext quick extraction
• Line 1219 — after ocrmypdf extraction
• Line 1501 — after multi-page PDF image OCR
• Line 1657 — after plain-text file reading

An identical copy lives in src/ocr/xml_extractor.rs and is applied at lines 850 (DOCX), 1051 (XLSX), and 1095 (legacy .doc via external tool).

A second, independent sanitizer — sanitize_text_for_db() in src/db_guardrails.rs — strips null bytes during the update_ocr_with_validation transaction path only.

Remaining gaps#

The database insert layer has no sanitization of its own:

The ingestion service also writes FailedDocument structs with raw error strings (e.g., from e.to_string()) directly into create_failed_document() 2 3 without sanitization. If an upstream error message or filename happens to contain a null byte — common with certain file-system paths or encoding bugs — the insert will fail.

The implicit contract is that all text reaching the DB has been sanitized by the OCR pipeline. That contract breaks for any non-OCR path (direct upload filenames, error message strings, metadata fields set before OCR runs).

failure_reason Constraint Mismatch#

The constraint#

The failed_documents table (introduced in migration 20250628000003) enforces an explicit CHECK constraint on failure_reason. The 17 allowed values are:

duplicate_content, duplicate_filename, unsupported_format, file_too_large, file_corrupted, access_denied, low_ocr_confidence, ocr_timeout, ocr_memory_limit, pdf_parsing_error, storage_quota_exceeded, network_error, permission_denied, virus_detected, invalid_structure, policy_violation, other

The same set is mirrored in ConstraintValidator::validate_failure_reason() in src/db/constraint_validation.rs.

The mismatch#

DocumentIngestionService::ingest_document() constructs FailedDocument records with two non-allowlisted values:

• "storage_error" — used when file_service.save_document_file() fails
• "database_error" — used when db.create_document() fails

When either of these branches executes, create_failed_document() 4 fires the INSERT, PostgreSQL rejects it with a CHECK constraint violation, and the error is caught by:

if let Err(failed_err) = self.db.create_failed_document(failed_document).await {
    warn!("Failed to create failed document record for storage error: {}", failed_err);
}

2 3

The warning is logged but execution continues. No failure record is persisted. Storage failures and database creation failures become invisible in the failed_documents table.

Why ConstraintValidator doesn't help#

ConstraintValidator::validate_failure_reason() exists for exactly this purpose but is never called in the ingestion code path before the insert. The validator also has a map_legacy_ocr_failure_reason() helper 5 that maps unrecognized strings to "other", which would serve as a valid fallback — but it too is unused here.

Fix options#

• Add "storage_error" and "database_error" to the CHECK constraint (requires a new migration).
• Map them to the existing "other" value at the call site, or use the existing ConstraintValidator::map_legacy_ocr_failure_reason() as a model.
• Call ConstraintValidator::validate_failure_reason() before every create_failed_document() call and substitute "other" on validation failure.

Key Files#