Embedding Pipeline#

Overview#

RAGFlow's embedding pipeline converts document chunks into dense vectors for storage in the search index. It spans two layers:

1. EmbeddingService — async orchestrator that drives batch encoding for a task, handles title/content weighting, and reports progress 1
2. embedding_model.py provider classes — synchronous, provider-specific adapters that implement the actual API calls 2

Batch Size Configuration#

The top-level batch size is read from settings.EMBEDDING_BATCH_SIZE, which defaults to 16 and can be overridden via the EMBEDDING_BATCH_SIZE environment variable 3. EmbeddingService.__init__ accepts an explicit embedding_batch_size override; if not supplied, it falls back to the setting 4.

EmbeddingService.embed_chunks Flow#

Entry point: EmbeddingService.embed_chunks(docs, embedding_model, parser_config)

1. Text extraction — EmbeddingUtils.prepare_texts_for_embedding(docs) returns (titles, contents). Content is pulled from question_kwd > content_with_weight; titles from docnm_kwd.
2. Title encoding — The first title is encoded once and tiled across all chunks (cost: 1 API call for the whole batch).
3. Content batch loop — Contents are sliced into chunks of _embedding_batch_size and dispatched via thread_pool_exec(_batch_encode_wrapper, ...). Each batch is rate-limited through ctx.embed_limiter 5.
4. Truncation — Before each content batch is sent to the model, every text is truncated to embedding_model.max_length - 10 tokens via truncate().
5. Vector assembly — EmbeddingUtils.stack_vectors() vstack-combines per-batch arrays; EmbeddingUtils.combine_title_content_vectors(tts, cnts, title_weight) applies a weighted blend (DEFAULT_TITLE_WEIGHT = 0.1, i.e. 10% title / 90% content).
6. Attachment — EmbeddingUtils.attach_vectors(docs, vects) writes each vector to the chunk dict under the key q_<N>_vec.
7. Return — (total_token_count, vector_size)

Provider-Level: Base._batched_encode#

The shared template for OpenAI-style providers is Base._batched_encode(texts, call_fn, *, batch_size, truncate_to=None). It:

• Optionally truncates each text to truncate_to tokens before issuing any calls.
• Runs a loop of ceil(len(texts) / batch_size) calls to the provider-supplied call_fn(batch) -> (embeddings, token_count) closure.
• Accumulates vectors into a single np.ndarray and sums token counts.
• Wraps any non-ModelException in a unified EmbeddingError, ensuring callers see a consistent exception type.

Response ordering is guaranteed by _sorted_by_index(), which sorts SDK result items by their .index attribute before extracting embeddings.

Provider Batch Sizes and Truncation Limits#

Sources: 6 7 8 9 10 11 12 13 14 15 16 17

DEFAULT_MAX_TOKENS = 8192 is the standard ceiling for most 8K-context providers 18.

DashScope URL Resolution#

QWenEmbed requires the DashScope native HTTP API (/api/v1) rather than the OpenAI-compatible path. _dashscope_native_http_api_url(base_url) detects known DashScope hostnames and maps them to the correct endpoint (international: dashscope-intl.aliyuncs.com, domestic: dashscope.aliyuncs.com). A context manager _dashscope_native_api_url_scope temporarily patches dashscope.base_http_api_url to minimize concurrency exposure.

Key Source Files#