Streaming Deduplication#

Overview#

LangBot implements anti-duplication logic across three layers to prevent duplicate or repeated streamed assistant output: the Dify runner (difysvapi.py), the WeCom adapter (wecombot.py), and the Feishu/Lark adapter (lark.py). Dify can emit multiple terminal events in a single stream, send both cumulative snapshots and delta chunks, and produce overlapping answer-node and message events. Platform adapters additionally guard against redundant card/message updates. Together these mechanisms ensure a single, deduplicated response reaches the user.

Dify Runner (difysvapi.py)#

_merge_stream_text — cumulative vs. delta detection#

All Dify streaming paths use _merge_stream_text(accumulated, incoming) as the core deduplication primitive. It handles two Dify stream styles:

• Cumulative snapshots: if incoming is longer than accumulated and starts with it, _merge_stream_text replaces the entire accumulator with incoming 1.
• Delta chunks: otherwise, it appends incoming to accumulated 2.

This logic prevents double-counting when Dify sends a full-answer snapshot after incremental deltas.

_chat_messages_chunk — yielded_final flag#

_chat_messages_chunk handles chatflow and chat application streaming. Dify can emit both workflow_finished and message_end events in the same stream, both of which set is_final = True. Without protection, the yield-every-8-chunks gate would emit the final chunk twice.

PR #2049 introduced a local yielded_final = False boolean that gates the yield block with not yielded_final and is set to True immediately after the first final chunk is emitted 3. This ensures subsequent iterations cannot re-yield the final chunk.

Workflow mode detection guard#

PR #2027 added workflow mode detection: in addition to workflow_started, any of node_started, node_finished, workflow_finished, or workflow_paused event types also triggers mode = 'workflow'. This defends against Dify deployments that omit the workflow_started event.

Answer-node replaces accumulator#

When mode == 'workflow' and a node_finished event carries node_type == 'answer', the extracted answer text replaces basic_mode_pending_chunk entirely rather than appending 4. The answer-node output is already the complete answer and would duplicate preceding message events.

The same pattern appears in _submit_workflow_form_blocking, where an answer_node_seen flag forces pending_content = answer when an answer node fires, discarding the prior accumulator.

_submit_workflow_form (streaming) — yield_this_iteration gate#

The streaming form-submission path uses a per-iteration yield_this_iteration = False flag. Only workflow_finished, workflow_paused, or a modulo-8 text/message chunk sets this flag to True 5 6 7 8, preventing the generator from emitting on every event.

WeCom Adapter (wecombot.py)#

Synthetic-event buffer deduplication#

When a Dify form-button click creates a "synthetic" event (no inbound message), _handle_synthetic_chunk buffers all non-final chunks and deduplicates on accumulation:

• If incoming content starts with previous, replace (cumulative snapshot) 9.
• If previous ends with content, keep (already included) 10.
• Otherwise, append 11.

The same prefix/suffix check runs again at flush time on the is_final chunk to reconcile buffered and final texts.

Feishu / Lark Adapter (lark.py)#

_lark_should_update_stream_element — throttle guard#

_lark_should_update_stream_element prevents every chunk from triggering a card update. The streaming card element (element_id: 'streaming_txt') is only updated when not resume_from and not form_data and (msg_seq % 8 == 0 or is_final) 12.

Cached-text comparison before card update#

Both normal streaming and resume-mode paths compare the new text_message against card_streaming_text[card_id] before issuing an API call. If the content is unchanged, the update is skipped entirely.

_lark_final_layout_texts — final card layout deduplication#

PR #2490 fixed a bug where the final streaming card would render the same reply text twice — once in the main streaming_txt element and again in the streaming_txt_resume placeholder. The bug appeared in non-resume rounds because both elements received the same accumulated text (resume_cached equaled text_message), duplicating the full reply.

_lark_final_layout_texts determines which text to render in each element:

• Non-resume round: return (text_message, '') — the full reply goes in the main element only; the resume placeholder remains empty 14.
• Resume round (Dify HITL): return (pre_pause_cached or text_message, resume_cached) — keep the pre-pause text in the main element and the resumed text in the placeholder, as they are distinct segments 15.

The function handles two edge cases:

• Empty pre-pause cache: when Dify paused before emitting any text, pre_pause_cached == '' is valid and must not fall back to text_message 16.
• Missing pre-pause cache: if pre_pause_cached is None (cache miss), fall back to text_message 17.

reply_message_chunk calls _lark_final_layout_texts on the final chunk and renders the returned tuple into the card layout.

Resume transition is single-entry#

The card_resume_transitioned set tracks which cards have already switched from "buttons visible" to "resume layout." The first resume chunk triggers the full layout transition and adds the card ID to the set 13; subsequent resume chunks go through a lighter text-diff path that only updates if text_message != cached.

Data Flow#

Key Source Files#

Related PRs#

• PR #2049 — Fix: yielded_final flag to deduplicate final chunk in Dify chatflow streaming
• PR #2027 — Refactor: Dify API output parsing, workflow mode detection and answer-node preference
• PR #2490 — Fix: stop duplicating final reply text in Lark streaming card