NDJSON Import System#

Overview#

The NDJSON import system handles bulk data ingestion for Sure via SureImport, the only import type that supports chunked, API-driven uploads. It is built around two core models — SureImport and ImportSession — and is exposed through the REST API at POST /api/v1/import_sessions 1.

The flow is two-phase: upload chunks → publish. The API controller is Api::V1::ImportSessionsController with three write actions: create, create_chunk, and publish 2.

Routes 3:

POST /api/v1/import_sessions
GET /api/v1/import_sessions/:id
POST /api/v1/import_sessions/:id/chunks → create_chunk
POST /api/v1/import_sessions/:id/publish

NDJSON Format#

Each line in the file is a JSON object with "type" and "data" keys. Validation checks only the first line 4. Supported entity types 5:

Account, Balance, Category, Tag, Merchant, RecurringTransaction, Transaction, Transfer, RejectedTransfer, Trade, Holding, Valuation, Budget, BudgetCategory, Rule

Allowed content types: application/x-ndjson, application/ndjson, application/json, application/octet-stream, text/plain 6.

Session Creation & Idempotency#

POST /api/v1/import_sessions calls ImportSession.create_or_find_for!, which implements idempotency via an optional client_session_id. If the same client_session_id is submitted again, the existing session is returned rather than creating a duplicate 7. If expected_chunks differs from the previously stored value, a ConflictError is raised 8.

The session has four statuses: pending, importing, complete, failed 9.

Chunked Upload#

Each chunk is uploaded via POST /api/v1/import_sessions/:id/chunks, calling attach_chunk!:

1. Sequence validation — sequence must be a positive integer and must not exceed expected_chunks if set 10.
2. SHA-256 checksum — computed on the raw content at upload time and stored on the SureImport record 11.
3. Idempotency — chunks can be re-uploaded. If the same sequence or client_chunk_id already exists, the system checks that the checksum matches; mismatches raise ConflictError 12.
4. Retry finalization — on re-upload, the file is re-attached only if the stored file is missing or its checksum differs 13.

Chunks are accepted only while the session is in pending or failed state; uploading to a session that is importing or complete raises a ConflictError 14.

Each chunk is stored as a SureImport record with an attached ndjson_file (Active Storage) 15.

Both a file multipart upload and a raw_file_content string body are accepted at the controller layer 16.

File Size & Row Count Limits#

Defaults defined in SureImport. Both are overridable at runtime via environment variables 17. The controller enforces the size limit per chunk before attaching 18. The row limit is enforced at publish time by summing rows_count across all chunks 19.

Publishing#

POST /api/v1/import_sessions/:id/publish calls publish_later:

1. sync_chunk_row_counts! — downloads each chunk and re-counts rows 20.
2. validate_publishable_chunks! — asserts at least one chunk exists, total rows ≤ limit, and that actual chunk sequences exactly match the declared expected_chunks range 21.
3. Transitions session to importing and enqueues ImportSessionJob on the :high_priority queue.

ImportSessionJob calls publish synchronously, which iterates chunks in sequence order, calling import! on each SureImport and delegating to Family::DataImporter. After all chunks succeed, status transitions to complete and a family sync is enqueued 22.

If any chunk fails, status rolls back to failed and error details are stored 23.

Readback Verification#

After each chunk is imported, SureImport records a readback verification: it snapshots entity counts before and after import, compares expected vs. actual deltas, and stores the result as not_verified → matched / mismatch / failed 24. The verification payload is accessible on the import record 25.

Key Files#