Vector Indexing and Embeddings#

Overview#

Dosu runs two parallel vector indexing pipelines: a production v1 pipeline backed by LanceDB and an experimental v2 pipeline backed by Turbopuffer. The two pipelines differ in vector store, embedding model, dimensions, and entity schema. V1 is the authoritative source of truth; v2 runs in shadow/dual-write mode, gated per-organization by the enable-indexing-v2 PostHog feature flag 1.

Pipeline Comparison#

V1 and v2 embeddings are incompatible — different dimensionality means they cannot be compared or substituted.

v1 Pipeline (LanceDB)#

Connection and Scoping#

LanceDBClient scopes each connection per data_source_id, mapping each to its own directory under LANCEDB_FILESTORE_URI . Both sync (connect) and async (connect_async) variants are supported.

Document Schema#

All v1 documents inherit from LDBDocument, sharing base fields: id, document_source_id, title, url, content, content_hash (MD5 upsert key), chunk/start_line/end_line, and created_at/updated_at. Every document stores a 1536-dimensional float32 embedding .

The 17 document types are registered in backend/core/db/lancedb/types/registry.py and organized across 12 modules in backend/core/db/lancedb/types/. Embedding model assignment per type is controlled by FIELD_MODEL_MAP in base.py:

Writes and Search#

Documents are upserted via merge_insert keying on content_hash (idempotent). Tables and FTS indexes are created lazily. Deletion targets document_source_id via SQL WHERE clause .

Vector search uses dot-product similarity with top_k=10 and similarity_threshold=0.5 defaults . Full-text search uses Tantivy/BM25 .

v2 Pipeline (Turbopuffer)#

Entity Families#

v2 consolidates the 17 v1 document types into 4 unified entity families 1:

Turbopuffer namespaces are keyed as {data_source_id}__{content_type} 1.

Chunk Pipeline#

Each entity flows through Load → Chunk → Embed → Store . Specialized chunk factories handle each entity type:

• change_request_chunk_factory.py
• knowledge_page_chunk_factory.py
• question_answer_chunk_factory.py
• thread_chunk_factory.py

All v2 chunks use Voyage voyage-4-large (1024 dimensions) 1. Upserts use replace_for_sources=[document_source_id] for atomic idempotent replacement .

Page Type Filtering#

FileLikeChunk and FileLikeDocument carry a page_type attribute — set only for Dosu-native pages; third-party synced pages carry page_type=None 1. PageFilters in backend/search_v2/pages/filters.py compiles this into an In clause for Turbopuffer attribute filtering.

Dual-Write Mechanics#

The dual-write is integrated in backend/workflows/index_batch/main.py. V2 runs first to prime the shared embedder cache (avoiding double-billing the embedding provider), then v1 runs. V2 errors are caught and logged; they never block v1 1.

V2 passes skip_mark_indexed=True so it does not update document_source.last_indexed_at_version — if it did, v1's loader would see a version match and short-circuit the batch with SkippedEntity 1. The _NoMarkDocumentSourceManager wrapper in v2_runner.py no-ops the mark_indexed() call to enforce this.

To promote v2 to GA: remove ENABLE_INDEXING_V2 from PosthogFeatureFlagKeys, drop all is_v2_enabled() guards, and deprecate v1 1.

Key Files#