Dataset Parsing Status#

Dataset parsing status tracks how far along each document in a knowledge base has progressed through RAGFlow's ingestion pipeline. The system exposes this data at two granularities: a per-dataset aggregated summary endpoint and a query parameter on the list-datasets endpoint.

Status Values#

Each document's run column in the Document table holds a numeric status. The five mapped values are 1:

"5" (SCHEDULE) is not mapped and is silently excluded from all aggregations.

SQL Aggregation: get_parsing_status_by_kb_ids#

The core aggregation logic lives in DocumentService.get_parsing_status_by_kb_ids (api/db/services/document_service.py). It accepts a list of kb_id strings and returns a nested dict keyed by kb_id, each with all five status fields pre-initialized to 0:

{
  "kb-abc": {"unstart_count": 10, "running_count": 2, "cancel_count": 0, "done_count": 15, "fail_count": 1},
  ...
}

The implementation issues a single Peewee ORM query with GROUP BY kb_id, run and a COUNT(id) aggregate, then maps the numeric run values to human-readable keys. Empty or missing kb_ids are always included with zeroed counts, ensuring consistent response shapes.

API Surface#

GET /api/v1/datasets/<dataset_id>/ingestions/summary#

The primary consumer of get_parsing_status_by_kb_ids is get_ingestion_summary in api/apps/services/dataset_api_service.py. It calls the aggregation for a single dataset and returns:

{
  "doc_num": 42,
  "chunk_num": 1023,
  "token_num": 190000,
  "status": {
    "unstart_count": 0,
    "running_count": 1,
    "cancel_count": 0,
    "done_count": 40,
    "fail_count": 1
  }
}

The route handler lives in dataset_api.py at GET /datasets/<dataset_id>/ingestions/summary.

GET /api/v1/datasets — include_parsing_status parameter#

The ListDatasetReq Pydantic model (in api/utils/validation_utils.py) defines:

class ListDatasetReq(BaseListReq):
    include_parsing_status: Annotated[bool, Field(default=False)]
    ext: Annotated[dict, Field(default={})]

This parameter was introduced alongside the feature added in PR #13481 ("feat: Support get aggregated parsing status to dataset via the API"). As of the current codebase, the parameter is accepted and validated at the API layer 2 but is not yet wired into the list_datasets service method 3 — it is present in the request schema but has no effect on the response. Use /ingestions/summary for per-dataset status.

Key Files#