Embedding Vector Validation#

RAGFlow's embedding pipeline enforces two invariants: (1) the number of vectors returned by an embedding provider must exactly match the number of input texts; (2) empty or whitespace-only inputs never reach a provider. Violations of these invariants silently corrupt the chunk→vector mapping, which causes incorrect retrieval at query time.

Pre-Processing: Filtering Empty Inputs#

Before any provider call, inputs are sanitized at two layers.

EmbeddingUtils._handle_whitespace() (shared utility, pipeline and task-executor paths) replaces any text that is empty or whitespace-only with the string "None" 1. This prevents providers from receiving blank strings, which some external APIs reject or return zero-length vectors for.

tokenizer.py _embedding() function (pipeline/canvas path) takes a stricter approach: it builds a valid_pairs list of (original_index, chunk) tuples, skipping any chunk whose concatenated, HTML-stripped text is empty 2. If all chunks are empty, it returns early without calling any provider 3. This was introduced in PR #14354 to replace the previous assert len(vects) == len(chunks) which would spuriously fail when some chunks had no embeddable content.

HTML table tags (<table>, <td>, <caption>, <tr>, <th>) are normalized to spaces before the whitespace check in both paths 4 5.

Count Validation: Vectors Must Match Inputs#

After embedding, both code paths explicitly check alignment:

• tokenizer.py asserts len(vects) == len(valid_pairs) 6 and then writes q_{dim}_vec only onto the chunks in valid_pairs, leaving non-embedded chunks untouched 7.

• EmbeddingService.embed_chunks() (task-executor path) asserts len(vects) == len(docs) after stacking all batch results 8. The upstream EmbeddingUtils.attach_vectors() raises a ValueError if the lengths diverge 9.

Provider-Level: Handling Out-of-Order Responses#

OpenAI-compatible endpoints (including LiteLLM proxies) may return embedding items in an order that does not match the input batch order. RAGFlow addresses this with:

• _sorted_by_index(items) — sorts SDK response objects by their .index attribute 10. Used by OpenAIEmbed, LocalAIEmbed, ZhipuEmbed, XinferenceEmbed, OpenRouterEmbed, and others 11.

• Base._openai_http_embeddings(response) — for providers using raw requests HTTP calls, sorts the data list by d.get("index", 0) 12. Used by NvidiaEmbed and SILICONFLOWEmbed.

Both strategies are stable no-ops when the provider returns items in order.

Provider-Level: Batching and Error Surface#

All OpenAI-style providers delegate to Base._batched_encode(), which:

• Optionally truncates each text to truncate_to tokens before sending.
• Issues ceil(len(texts) / batch_size) calls, accumulating vectors and token counts across batches.
• Wraps any provider exception in a single EmbeddingError 13, ensuring callers see a consistent exception type regardless of which SDK failed.

The standard token ceiling for most providers is DEFAULT_MAX_TOKENS = 8192 14. Provider-specific overrides exist (e.g., Zhipu embedding-2: 512 tokens; Zhipu embedding-3: 3072 tokens 15).

Known Edge Cases and Issues#

The BaiduYiyanEmbed 2-D shape bug illustrates the adapter contract: encode_queries must return a 1-D vector of shape (D,), while encode() must return shape (N, D) 16. Any provider that returns the wrong shape will misalign downstream consumers silently.

Key Source Files#