Dataset Item Processing Pipeline#

Dataset items flow from API input through normalization, schema validation, and finally into Prisma/PostgreSQL. The pipeline is shared by the Public API and tRPC (internal UI) and is implemented in two layers: the repository (dataset-items.ts) and the validator (DatasetItemValidator).

Entry Points#

The Public API request body is typed by PostDatasetItemsV1Body, where input, expectedOutput, and metadata are all z.any().nullish() — no coercion at the Zod layer; all normalization happens downstream.

Step 1 — Normalization (DatasetItemValidator.normalize)#

DatasetItemValidator is instantiated once per write operation (or once per dataset in bulk) so Ajv schemas are compiled exactly once 2. The private normalize method handles three concerns:

1. String coercion — If the incoming value is a string, it is parsed as JSON via parseJsonPrioritised 3. This covers tRPC callers that serialize values to JSON strings before sending. If the string is not valid JSON, it is stored as a plain string. The Public API sends already-parsed objects and takes the else branch directly.

2. Unsafe integer preservation — parseJsonPrioritised uses lossless-json to detect numbers with 13+ digits or scientific notation. Safe numbers become JS Number; unsafe integers (e.g. 107505301260286111) are preserved as strings to avoid IEEE-754 precision loss 4. This behavior was introduced in PR #14119.

3. Control character sanitization — When normalizeOpts.sanitizeControlChars is true (always set by the Public API), sanitizeJsonValue recursively strips C0/C1 control characters from all string leaves 5. PostgreSQL TEXT columns reject \u0000; \n, \t, and \r are preserved.

Empty string maps to null; undefined passes through as undefined 6.

Step 2 — Schema Validation (DatasetItemValidator.validateAndNormalize)#

After normalization, validateAndNormalize:

1. Create guard — When validateOpts.normalizeUndefinedToNull is true (create operations), null or undefined input is rejected immediately with a PayloadError 7.
2. Schema validation — Delegates to DatasetSchemaValidator which holds Ajv ValidateFunction instances compiled from the dataset's inputSchema / expectedOutputSchema. Missing schemas are treated as valid 8. A single DatasetSchemaValidator instance is reused across all items in a batch, giving a 3800×+ speedup versus per-item Ajv compilation 9.
3. Prisma value conversion — null → Prisma.DbNull (explicit DB NULL); undefined → undefined (field skipped in partial updates) 10.

On failure, the method returns a PayloadError containing field-level inputErrors and expectedOutputErrors arrays 11.

Step 3 — Prisma Storage#

Validated payloads are written as Prisma.InputJsonValue | Prisma.NullTypes.DbNull | undefined, matching Prisma's JSON column types 12. The repository supports two write strategies via executeWithDatasetServiceStrategy:

• STATEFUL — Direct prisma.datasetItem.create/update. 13
• VERSIONED — Temporal table: invalidates the current row (validTo = newValidFrom) then inserts a new row 14. Reads use valid_to IS NULL or point-in-time valid_from ≤ T < valid_to queries 15.

Bulk inserts under VERSIONED mode run in a single transaction: one updateMany to invalidate, then createMany for all new rows 16.

Key Files#