Dataset Segment Batch Import#

Batch import lets users add many segments to a knowledge-base document at once by uploading a CSV file. The operation is asynchronous: the API returns a job ID immediately, and callers poll a status endpoint until the job reaches a terminal state.

Entry Points#

Job Lifecycle (Redis)#

The controller and task coordinate via a single Redis key: segment_batch_import_{job_id} 1.

1. waiting — written by the controller with setnx (no TTL) immediately before dispatching the Celery task 2. Because setnx is used, the write is a no-op if the key already exists, which guards against accidental double-dispatch.
2. error — written by the task with setex + 600-second TTL if any exception occurs 3.
3. completed — written by the task with setex + 600-second TTL after vector creation succeeds 4.

The 600-second TTL on terminal states ensures automatic Redis cleanup; there is no explicit deletion step.

Task Processing Phases#

batch_create_segment_to_index_task runs in four sequential phases 5:

1. Setup (session 1) — validates dataset, document, and upload file from the DB; extracts the storage key and configuration dicts; any failure sets status error and returns 6.
2. CSV parsing — downloads the file from object storage into a temp directory, reads it with pandas, and builds a list of {content, answer?} dicts. QA-indexed datasets expect two columns; plain datasets expect one 7.
3. Segment creation (session 2) — inserts one DocumentSegment row per CSV row, with status=COMPLETED and indexing_at/completed_at timestamps set immediately 8. Token counts come from the embedding model for high-quality datasets, or default to 0 for economy datasets 9.
4. Vectorization (session 3) — calls VectorService.create_segments_vector() to push the new segments to the vector store.

Error Handling and the Pre-Fix Bug#

Prior to PR #38863, only the setup phase had error handling. If CSV parsing, DB writes, or vector creation failed, the Redis key stayed at waiting (no TTL, no expiry). The status endpoint would return waiting indefinitely, causing the frontend to poll forever without surfacing the error 10.

The fix wraps the entire processing phase (phases 2–4) in a try/except block that calls redis_client.setex(indexing_cache_key, 600, "error") on any exception 10. Two concrete failure modes are covered by tests:

• Empty CSV (header row only) — raises ValueError("The CSV file is empty.") 11.
• QA dataset with single-column CSV — raises IndexError on row.iloc[1] 10.

Both now correctly resolve to status error rather than stalling at waiting.

Key Files#