Weaviate Vector Store#

Overview#

WeaviateVector is Dify's Weaviate adapter, implementing the BaseVector abstract interface for RAG vector storage. The implementation lives at api/providers/vdb/vdb-weaviate/src/dify_vdb_weaviate/weaviate_vector.py. Configuration is driven by WeaviateConfig in api/configs/middleware/vdb/weaviate_config.py:

The adapter uses a process-singleton Weaviate client with double-checked locking and registers an atexit handler for graceful teardown 1.

UUID Generation: History and Current State#

Old Behavior (Bug — now fixed)#

_get_uuids() previously derived each object's Weaviate UUID as uuid5(URL_NAMESPACE, page_content). This caused two problems:

1. Content collision: Two documents containing identical text produced the same UUID. The second write silently overwrote the first, corrupting metadata and breaking metadata-based filters. 2
2. Deletion mismatch: Dify's delete_by_ids() receives the segment's index_node_id (a UUID4 logical key) as the IDs to delete. Since Weaviate objects were keyed by content-derived UUID5, the IDs never matched — every delete call returned 404, which was swallowed, leaving orphaned vectors. 3

PR #36140 proposed including doc_id in the UUID5 hash to prevent collisions, but did not resolve the deletion-side mismatch.

Current Behavior (Fixed)#

_get_uuids() now uses doc.metadata["doc_id"] directly as the Weaviate object UUID, matching the BaseVector._get_uuids() contract. doc_id is the segment's index_node_id — the same value delete_by_ids() receives, so write and delete paths are now consistent by construction.

A random uuid4() fallback is used only when doc_id is absent (e.g., test fixtures), noted in code comments as acceptable because such objects are not expected to go through the cleanup path 4.

This fix was introduced by PR #39176.

Deletion Patterns: UUID-based vs. Metadata-based#

The Core Deletion Bug (now fixed)#

Prior to the fix, delete_by_ids() called col.data.delete_by_id(uid) for each uid, treating logical index_node_id values as Weaviate object UUIDs. Since UUIDs were content-derived (UUID5), this always 404'd and was silently swallowed 5. The cleanup task reported success while no vectors were removed.

Measured impact (issue #40457): Deleting 4 documents left 9,165 orphan objects in one knowledge base. Orphans are returned by vector search but then dropped by Dify (no matching Postgres row), silently consuming top_k retrieval slots and degrading search quality over time 6.

Current delete_by_ids() — Dual-Path 7#

1. UUID-path (new objects): col.data.delete_by_id(uid) for each ID. Hits objects written after the UUID fix where Weaviate UUID == doc_id. 404s are suppressed — expected for legacy pre-fix objects.
2. Metadata-path (legacy fallback): col.data.delete_many(where=Filter.by_property("doc_id").contains_any(ids)) — catches any object whose doc_id property matches, regardless of what UUID was used at write time. This backward-compatible sweep ensures pre-fix orphans are reaped on the next deletion pass.

Other Deletion Methods#

delete_by_metadata_field and delete were never affected by the UUID mismatch. delete drops the entire collection and is the path taken by clean_dataset_task when a full dataset is removed.

Collection Schema, Caching, and Query Resilience#

Schema#

Collections are created with these properties 8:

• text — chunk content (tokenized per WEAVIATE_TOKENIZATION)
• document_id — dataset document ID (used for document_ids_filter in search)
• doc_id — segment index_node_id; the deletion key
• doc_type, chunk_index, is_summary, original_chunk_id — parent-child indexing metadata

_ensure_properties() backfills any missing properties on existing collections 9, enabling schema evolution without data loss.

Race Condition Prevention#

Collection creation acquires a Redis distributed lock (vector_indexing_lock_{collection_name}, 20 s timeout) and sets a TTL-1h cache key (vector_indexing_{collection_name}) to skip redundant existence checks on subsequent calls 10.

Query Retry on Schema Mismatch#

Both search_by_vector and search_by_full_text catch WeaviateQueryError, call _ensure_properties(), and retry the query once 11. This handles the case where a collection was created by an older Dify version that lacked newer schema properties.

Weaviate 1.39 Upgrade and Docker Registry Migration#

PR #38214 upgraded the bundled self-hosted Weaviate from 1.27.0 to 1.39.0 (currently 1.39.2 in docker-compose.yaml) and changed the image source from Docker Hub to Weaviate's own registry 12:

cr.weaviate.io/semitechnologies/weaviate:1.39.2

Self-hosted impact: Environments with firewall or network rules that allow docker.io but block cr.weaviate.io will fail to pull the image on next docker compose pull. Update egress rules accordingly.

Upgrade path from 1.27: Increment through each minor version with graceful stops between upgrades to avoid HNSW commit log truncation issues. Version 1.39 itself has no breaking API changes and is compatible with weaviate-client 4.x.

WEAVIATE_GRPC_ENABLED was removed (PR #33378); gRPC is always active and the endpoint is configured via WEAVIATE_GRPC_ENDPOINT 13.