Indexing Failure Recovery#

When pqa index (or get_directory_index) processes a directory of papers, failures can occur at two distinct layers: PDF/text parsing and metadata fetching. The recovery mechanisms at each layer differ significantly.

Failure Marking and Skip-on-Rerun#

The core persistence primitive is FAILED_DOCUMENT_ADD_ID = "ERROR", a sentinel stored in the SearchIndex.index_files dictionary (a zlib-compressed pickle at {index_directory}/{index_name}/files.zip) in place of a real content hash.

When any exception escapes process_file, the file is immediately marked and the index is flushed to disk 1:

await search_index.mark_failed_document(file_location)
await search_index.save_index()

mark_failed_document writes "ERROR" as the file's hash. filecheck returns True for any entry in index_files regardless of value, so files marked "ERROR" are permanently skipped on subsequent runs unless the index is manually cleared or rebuilt.

⚠️ Known limitation (open issue): Once a file is marked ERROR, later index runs silently skip it even when the original failure was transient (e.g., a 429 rate-limit). There is currently no --reprocess-failed CLI flag or IndexSettings field to retry them. See issue #1275 for context.

Parse/Add Failures (ValueError, ImpossibleParsingError)#

process_file catches all exceptions from Docs.aadd(), but only swallows ValueError and ImpossibleParsingError 2:

if not isinstance(e, ValueError | ImpossibleParsingError):
    raise

• ImpossibleParsingError — raised when a document is fundamentally unparseable (corrupt/unsupported format). Swallowed; the file is marked ERROR and the run continues.
• ValueError — also swallowed (rationale is marked # TODOC in the source).
• Any other exception — re-raised into the anyio.TaskGroup, which cancels the entire get_directory_index run.

Metadata Fetch Failures (HTTP 4xx/5xx, Timeouts, Retries)#

Metadata lookup for DOIs and titles goes through DOIOrTitleBasedProvider.query, which catches and suppresses multiple failure modes before they can reach process_file:

When a provider returns None, the document is indexed with a plain Doc (no enriched metadata) rather than failing.

Transient HTTP retries are handled upstream in _get_with_retrying, which uses tenacity with up to 5 attempts and incremental backoff for:

• aiohttp.ServerDisconnectedError
• aiohttp.ClientConnectionResetError
• aiohttp.ClientResponseError with status 500 or 504

As of the current codebase (is_retryable), HTTP 429 (Too Many Requests) is not in the retryable set, meaning a 429 from Semantic Scholar escapes to DOIOrTitleBasedProvider.query — where it is caught as aiohttp.ClientResponseError and returns None. PR #1353 (open as of this writing) tracks a related bug where a migration from aiohttp to httpx briefly caused 429s to propagate all the way up and crash index runs.

Concurrency note: IndexSettings.concurrency (default 5, low intentionally for users without API keys) 4 controls the semaphore over parallel process_file calls. Reducing this value decreases the rate of outbound metadata requests and can reduce 429 frequency. Setting SEMANTIC_SCHOLAR_API_KEY / CROSSREF_MAILTO environment variables raises provider rate limits.

Key Source Files#