Table Reader Error Handling#

The table reader implements a two-stage batch fetch pipeline for backfills: a primary-key fetch (fetch_batch_pks) followed by a full-row fetch (fetch_batch). These two stages handle query errors differently, creating an important asymmetry that affects fault-tolerance and observability.

Key files:

• lib/sequin/runtime/table_reader.ex — core query functions
• lib/sequin/runtime/table_reader_server.ex — GenStateMachine server that orchestrates fetches and handles errors

Two-Stage Fetch Pipeline#

Both fetch stages run inside Task.Supervisor.async_nolink/2 tasks 1 and return results to the GenStateMachine as {:task_result, res}.

1. ID fetch — fetch_batch_pks/4 selects only primary key and cursor columns, returns {:ok, %{pks: [...], next_cursor: ...}}. The server stores the PKs in an ETS multiset keyed by batch_id 2.

2. Batch fetch — fetch_batch/6 fetches full rows, wraps them in Message structs, and emits a high-watermark logical message via with_watermark/6.

The with_watermark wrapper uses a safe with chain for its own Postgres calls 3. Errors from either stage propagate back as {:error, ...} through {:task_result, res} — unless a crash occurs first.

The Core Inconsistency: Safe vs. Unsafe Query Patterns#

fetch_batch_pks and fetch_batch handle Postgres query errors differently:

fetch_batch_pks — bare match (unsafe):
Inside the with :ok <- validate_primary_key_columns(...) block, the query is executed with a bare match 4:

{:ok, %Postgrex.Result{} = result} = Postgres.query(db_or_conn, sql, params, timeout: timeout)

If Postgres.query/4 returns {:error, ...}, this raises a MatchError, crashing the task process. The GenServer only sees a :DOWN message, not a structured {:error, error} tuple.

fetch_batch — case statement (safe):
The query is wrapped in case 5:

case Postgres.query(db_or_conn, sql, params, timeout: timeout) do
  {:ok, ...} -> ...
  error -> error
end

Errors are returned as {:error, error} through {:task_result, res}, enabling full error classification in the server.

All other Postgres.query call sites in the same file use with chains and are safe: fetch_first_row, fast_count_estimate, and fetch_slot_lsn. The bare match in fetch_batch_pks is an outlier.

Server-Side Error Handling#

ID Fetch Errors (fetch_batch_pks path)#

When the task returns {:task_result, {:error, error}} — currently only reachable if validate_primary_key_columns fails, not from the query itself due to the bare match — the server at 6 applies differentiated handling:

Important: Because the query at line 177 uses a bare match, a Postgres query failure bypasses this entire handler. The task crashes, and the server only receives a :DOWN message.

Batch Fetch Errors (fetch_batch path)#

Handled at 7. Timeout errors trigger page-size reduction without incrementing failure count; all other errors increment successive_failure_count. There is no Postgrex-specific or invariant-specific handling here (unlike the ID fetch path).

Task Crash () Path#

Both ID fetch task crashes and batch fetch task crashes are caught via {:DOWN, ref, :process, _pid, reason} handlers. Both simply log the reason and increment successive_failure_count — no differentiation by error type, no health event emission.

When a fetch_batch_pks query fails, the actual error path is: query raises MatchError → task process dies → :DOWN handler runs → generic retry. The richer handler at lines 506–540 is not invoked.

Impact and Fix#

The bare match in fetch_batch_pks 8 has three practical consequences:

1. Lost error classification — timeout vs. Postgrex error vs. transient error all look identical to the :DOWN handler; the page-size optimizer never learns about timeouts from this stage.
2. No health events — Postgrex.Error cases that would normally emit a backfill_fetch_batch health warning are silently swallowed.
3. No graceful no_primary_key_columns handling — the invariant check runs before the bare match, so this specific case is still handled correctly; but any future invariant checks added after the query would not be.

Fix: Replace the bare match with a case expression matching the pattern used in fetch_batch 5, so errors propagate as {:error, error} through {:task_result, res} and reach the classification logic at lines 506–540.