Dataset Segmentation Configuration#

Segmentation (chunking) is configured during Step 2 of knowledge base document creation. It controls how raw documents are split into indexable chunks before being embedded and stored. The mode and parameters chosen here directly determine chunk granularity, retrieval quality, and the shape of the indexing pipeline.

Key entry points:

• UI: step-two/index.tsx — renders mode-specific option cards
• Backend runner: IndexingRunner.run() — drives the Extract → Clean → Segment → Index pipeline 1
• Celery task: _document_indexing() — async entry point 1

Chunking Modes#

Three modes are defined by the ChunkingMode enum 2, which maps to the doc_form API field 3:

Note: The QA mode is only available on Community/Enterprise (non-cloud) editions 4.

General Mode#

Configured via GeneralChunkingOptions 5. Exposes three core fields rendered by inputs.tsx 6:

• Separator (segmentIdentifier) — delimiter string; supports escape sequences (e.g. \n\n). Rendered via DelimiterInput 7.
• Max chunk length (maxChunkLength) — upper-bounded by NEXT_PUBLIC_INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH env var; minimum 1 8.
• Overlap (overlap) — characters of overlap between adjacent chunks; minimum 1 9.

Automatic mode defaults (set by DatasetProcessRule.AUTOMATIC_RULES): delimiter \n, max tokens 500, chunk overlap 50 10.

Parent-Child Mode#

Configured via parent-child-options.tsx 11. Offers two parent modes selected via RadioGroup 12:

The ParentMode type is a string union 'full-doc' | 'paragraph' 13. Child chunk settings (delimiter, max length) are configured separately from parent settings.

Backend: Text Splitter Selection#

_get_splitter() in index_processor_base.py selects the splitter based on process_rule.mode 14:

The max_tokens parameter is validated to be between 50 and INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH before being passed as chunk_size 15.

FixedRecursiveCharacterTextSplitter#

Defined in api/core/rag/splitter/fixed_text_splitter.py 16. Behavior:

1. Split by the user-defined fixed_separator first (escape sequences decoded via codecs.decode).
2. Chunks exceeding chunk_size are recursively split using a fallback separator chain: ["\n\n", "\n", "。", ". ", " ", ""] — which covers CJK punctuation and space for multilingual text.

Previously Known Issue (Fixed in PR #39498): When recursive splitting fell through to the space (" ") separator, re.split(r" +", text) stripped the spaces, and the keep_separator restoration ran only in the non-space branch. Merged chunks then lost word-boundary spaces. Affected Korean, and any language relying on spaces when chunks exceeded size limits. Issue #39403 17 was resolved by PR #39498, which corrected the indentation so that the keep_separator logic applies to all separators including spaces. Spaces are now correctly preserved during recursive splitting when keep_separator is enabled.

Indexing Pipeline Effect#

After segmentation, chunks flow into the index stage via IndexingRunner._load() 1:

• high_quality — chunks are embedded via the dataset's embedding model and written to the vector store (parallelized across 10 threads).
• economy — keyword index built with Jieba; no embeddings.

Parent-Child mode diverges at the index processor level. ParentChildIndexProcessor only embeds and stores child chunks in the vector store; parent chunks are stored in document_segments unembedded. At query time, child doc_ids are looked up → ChildChunk.segment_id → parent DocumentSegment returned to the LLM 18.

Known bug (open as of 2026-07-25): Token counting during indexing sends all chunks to the embedding plugin in a single unbatched request, unlike the embedding step which batches by MAX_CHUNKS. For large documents (thousands of chunks), this can trigger HTTP 413 errors or OOM-kill the plugin daemon. Tracked in #39560 19. The embedding call itself is already correctly batched in cached_embedding.py.

Pre-Processing Rules#

Applied before splitting by CleanProcessor.clean() 1. Configurable in the UI via checkboxes in GeneralChunkingOptions 20:

API Configuration Reference#

When creating documents via the Service API, segmentation is controlled through the process_rule field of the request payload 3:

{
  "doc_form": "text_model", // or "hierarchical_model"
  "indexing_technique": "high_quality",
  "process_rule": {
    "mode": "custom", // "automatic" | "custom" | "hierarchical"
    "rules": {
      "pre_processing_rules": [
        { "id": "remove_extra_spaces", "enabled": true },
        { "id": "remove_urls_emails", "enabled": false }
      ],
      "segmentation": {
        "delimiter": "\n\n",
        "max_tokens": 500,
        "chunk_overlap": 50
      }
    }
  }
}

If process_rule is omitted, the dataset's most recent process rule is reused. For hierarchical mode, set doc_form: "hierarchical_model" and include parent/child-specific segmentation config. Indexing is asynchronous — poll GET /datasets/{dataset_id}/documents/{batch}/indexing-status for progress through stages: waiting → parsing → cleaning → splitting → indexing → completed 3.