Knowledge Graph Extraction Evaluation#

Docling-graph provides two complementary evaluation layers for measuring knowledge graph extraction quality:

1. Template-level evaluation (docling_graph/templategen/evaluate.py) — runs real pipeline extractions and reports grounding precision, attribute fill-rates, and graph audit signals. No ground truth required; advisory only.
2. Ground-truth comparison (docs/examples/scripts/16_extraction_evaluation.py) — scores an extracted graph.json against a reference JSON, computing node/edge P/R/F1, attribute completeness, and integrity checks. Used as a regression harness when tuning templates or extraction contracts.
3. Benchmark aggregation (docs/examples/scripts/17_benchmark_aggregate.py) — aggregates eval.json files across a run matrix (documents × contracts × LLM serialization formats × repeats) into a single comparison table.

Metrics Reference#

Node and Edge P/R/F1#

The evaluate() function in Example 16 computes micro precision, recall, and F1 separately for nodes and edges — never averaged together 1. TP/FP/FN are summed across all classes before computing ratios, so large classes dominate 2.

Per-entity-class metrics are computed first, then the micro summary is derived 3.

Three matching tiers determine whether an extracted node counts as a TP 4:

Structural alignment is activated automatically when the GT identifier is not found in the source text (--structural-align auto, the default) 5. It can be forced with --structural-align on or disabled with off.

Edge metrics follow the same strict/relaxed/aligned tiers: edge endpoints are resolved through the node alias map before comparing (label, source, target) triples 6.

The prf() helper computes per-class scores; micro_prf() sums TP/FP/FN across classes.

Grounding Precision#

Template-level grounding (in evaluate.py) checks whether extracted string node-attribute values appear as whitespace-normalized substrings within ProvenanceLedger chunk texts 7. The GroundingReport model exposes:

• checked / grounded counts
• precision = grounded / checked 8
• root_id_synthetic flag — set when the root identity was filled with the file-stem fallback injected by repair_root_identity, not actual document text 9
• ungrounded_samples — up to 10 Class.field=value samples not found in any chunk 10

Only string fields with ≥ 2 characters are checked; numeric/date values are excluded because they reformat freely 11.

The aggregate EvaluationSummary rolls up grounding_precision as grounded / checked across all documents 12.

Ground-truth verbatim ratio (Example 16) is a complementary check on matched nodes: long string fields (≥ 80 chars) in the extracted graph are tested as normalized substrings of the served source text. The result is reported as verbatim_ratio 13.

Attribute Completeness#

Completeness measures the fill-rate of declared fields on matched nodes — how many expected attributes were actually populated by the extractor.

• In the ground-truth comparison (Example 16): computed on strictly-matched nodes only; reports filled/expected per field per class 14. The benchmark aggregator rolls this up to a single completeness ratio 15.
• In the template evaluator: ClassFillRate tracks fill_rate as the mean fraction of declared node attributes filled per node, per class. The mean_fill_rate in EvaluationSummary averages across all classes and documents .

Graph Integrity Signals#

Both evaluation paths surface integrity issues that can mask inflated F1 scores:

These are reported under the integrity key in eval.json 18 and translated to rulebook clauses in the template evaluator via AUDIT_KEY_RULEBOOK.

Edge fan-out is an additional diagnostic: per edge label, the share of edges hanging off the single busiest source node. High fan-out in the extracted graph (≥ 0.6) with low fan-out in ground truth (≤ 0.4) flags "dump everything on one parent" pathology .

Entry Points#

Typical workflow:

# Step 1 – per-run scoring (produces eval.json)
uv run python docs/examples/scripts/16_extraction_evaluation.py \
    --graph outputs/RUN_DIR/docling_graph/graph.json \
    --truth data/insurance_terms/ground_truth.json \
    --template docs.examples.templates.insurance_terms.AssuranceMRH \
    --source document.md \
    --out eval.json

# Step 2 – cross-run aggregation
uv run python docs/examples/scripts/17_benchmark_aggregate.py \
    --root outputs/benchmarks

21 22

Key Design Decisions#

• Domain-agnostic: any (template, ground_truth.json, graph.json) triple is valid — the same harness works for insurance, biomedical, legal, or any other domain 23.
• Structural alignment for synthesized IDs: when a ground-truth author invents identifier slugs (e.g. STUDY-LFP-GELATION), strict identity matching is unwinnable by design. The evaluator auto-detects this via source-text presence checks and falls back to attribute-overlap pairing 24.
• No LLM judging: the template evaluator reports only signals the converter already emits — audit keys, fill-rates, grounding — without any additional LLM calls 25.
• Aligned rung is the fair headline number: the benchmark synthesis report treats the aligned F1 as the primary comparison metric when identity drift is present 26.