Audio and Video Processing#

Docling converts audio and video files into a DoclingDocument — the same intermediate representation used for PDFs and DOCX files — that can then be exported to Markdown, JSON, HTML, or DocTags and fed into RAG pipelines, summarizers, or search indexes.

Audio files run through the ASR pipeline (AsrPipeline), which transcribes speech to text. Video files run through the video pipeline (VideoPipeline), which additionally samples representative frames and optionally assigns speaker labels via diarization 1.

Supported formats 2:

Prerequisites: Both pipelines require ffmpeg on your PATH for audio decoding. Install with brew install ffmpeg (macOS), apt-get install ffmpeg (Linux), or winget install ffmpeg (Windows) 3.

Installation#

ASR is an optional extra 4:

pip install "docling[asr]"

For video with frame sampling and diarization 5:

pip install "docling-slim[format-video]"

format-video includes everything from asr plus resemblyzer, soundfile, scikit-learn, and librosa.

Pipelines and Entry Points#

ASR Pipeline (audio)#

The primary classes:

• AsrPipeline — orchestrates transcription for audio inputs
• AsrPipelineOptions — top-level configuration; its asr_options field accepts any InlineAsrOptions preset
• asr_model_specs — preset constants (WHISPER_TINY, WHISPER_TURBO, etc.) and the AsrModelType enum used by the CLI

Minimal setup 6:

from docling.datamodel import asr_model_specs
from docling.datamodel.pipeline_options import AsrPipelineOptions
from docling.document_converter import AudioFormatOption, DocumentConverter
from docling.pipeline.asr_pipeline import AsrPipeline

pipeline_options = AsrPipelineOptions()
pipeline_options.asr_options = asr_model_specs.WHISPER_TURBO

converter = DocumentConverter(
    format_options={InputFormat.AUDIO: AudioFormatOption(pipeline_cls=AsrPipeline, pipeline_options=pipeline_options)}
)
result = converter.convert(Path("recording.mp3"))
print(result.document.export_to_markdown())

The output is paragraph-level Markdown with per-segment timestamps: [time: 0.0-4.0] Transcribed text here 7.

Video Pipeline#

VideoPipeline orchestrates three steps 8:

1. Extract audio from the video via ffmpeg (to a 16kHz mono WAV) and transcribe it using the shared ASR transcriber
2. Sample representative frames using fixed-interval or scene-change sampling
3. Merge transcript segments and frames by timestamp into a single DoclingDocument

Frame images and transcript text are interleaved chronologically in the output document 9. Configuration lives in VideoPipelineOptions.

Frame sampling modes 10:

Set generate_frame_images=False to transcribe only, or max_sampled_frames to cap the total frame count.

Speaker diarization: set enable_diarization=True in VideoPipelineOptions to attribute transcript segments to speakers via Resemblyzer embedding clustering. The number of speakers is auto-detected. If the required packages are absent, diarization is silently skipped and transcription proceeds normally 11 12.

CLI 13:

docling --to md video.mp4 # fixed-interval (default)
docling --to md --video-sampling-mode scene --video-prominence 0.03 video.mp4 # scene-change
docling --to md --video-sampling-mode scene --video-diarization video.mp4 # with diarization

Backend Selection and ASR Models#

Three interchangeable ASR backends are available 14:

Auto-selecting presets#

Presets like WHISPER_TURBO, WHISPER_LARGE, etc. auto-select the backend at import time by probing for MPS availability and the mlx-whisper package. Priority order 15:

1. MLX Whisper — on Apple Silicon when mlx-whisper is installed
2. Native Whisper — everywhere else

WhisperS2T is never auto-selected; use explicit _S2T presets to opt in.

Forcing a backend#

Use suffixed presets to pin a backend regardless of hardware 16:

• WHISPER_TURBO_NATIVE — force native Whisper (CPU/CUDA)
• WHISPER_TURBO_MLX — force MLX (Apple Silicon)
• WHISPER_LARGE_V3_S2T — force WhisperS2T (CPU/CUDA only, not available on Apple Silicon)

The full preset list is in asr_model_specs.py. The CLI uses --asr-model <preset_name> (lower-cased) for both audio and video inputs 17.

Key option classes#

All defined in pipeline_options_asr_model.py:

Shared base fields on InlineAsrOptions: repo_id, timestamps (default True), temperature (default 0.0), max_time_chunk (default 30s), torch_dtype.

Language: Native and MLX options default language=None (auto-detect from the first 30 seconds). WhisperS2T defaults to "en". Override with an ISO 639-1 code when needed 18.

Key Source Files#

Limitations#

19

See Also#

• Official usage guide — installation, RAG examples, CLI reference
• Video pipeline notebook — end-to-end example
• Apple Silicon support details — MPS vs MLX decision matrix across all Docling components