SpanGroupIndexer._ensure_disjoint rejects overlapping predicted spans — ~28k prod failures/30d

[gas2own]

Summary

Document.annotate(...) fails whenever a predictor emits overlapping SpanGroups, because SpanGroupIndexer._ensure_disjoint() treats any overlap as fatal:

ValueError: Detected overlap with existing SpanGroup(s) [(35860, 36128, 11), (36124, 36125, 12)] for <mmda.types.annotation.SpanGroup ...>

In production this is the 2nd-largest mmda failure family: ~28,000 failures over the last 30 days, across three TIMO services that all call doc.annotate(...) with model-predicted spans.

Evidence (production, last 30 days)

Representative traceback (bibentry-detector-v0):

File ".../ai2_internal/bibentry_detection_predictor/interface.py", line 106, in predict_one
    doc.annotate(bib_entries=bib_entry_span_groups)
File ".../mmda/types/document.py", line 148, in _annotate_span_group
    self.__indexers[field_name] = SpanGroupIndexer(span_groups)
File ".../mmda/types/indexers.py", line 62, in __init__
    self._ensure_disjoint()
File ".../mmda/types/indexers.py", line 73, in _ensure_disjoint
    raise ValueError(
ValueError: Detected overlap with existing SpanGroup(s) ...

service	model_package	failures (30d)
bibentry-detector-v0	mmda	~27,925
citation-mentions-v0	mmda	85
citation-links-v0	mmda	18
total		~28,028

Source: timo_services.invocations Athena table, window 2026-05-10 → 2026-06-09.

How to regather (Athena CLI)

aws athena start-query-execution \
  --query-execution-context Database=timo_services \
  --result-configuration OutputLocation=s3://<your-athena-results-bucket>/ \
  --query-string "
SELECT service, COUNT(*) AS failures
FROM timo_services.invocations
WHERE model_package='mmda' AND outcome='failure'
  AND error LIKE '%Detected overlap with existing SpanGroup%'
  AND year=2026 AND month IN (5,6)
  AND timestamp >= to_unixtime(current_timestamp - interval '30' day)
GROUP BY service ORDER BY failures DESC"
# then: aws athena get-query-results --query-execution-id <id> --output table

Suggested fix

A model that proposes two spans that happen to overlap (extremely common for bib-entry / citation detectors that operate per-region) should not crash the entire document annotation. The indexer should tolerate overlap rather than raise. Options:

Option A — opt-in non-disjoint indexing. Let callers that expect overlap pass a flag, so existing strict callers are unaffected:

# indexers.py
class SpanGroupIndexer:
    def __init__(self, span_groups, allow_overlap: bool = False):
        ...
        if not allow_overlap:
            self._ensure_disjoint()

# document.py — thread the flag through annotate()/_annotate_span_group()
def annotate(self, *, allow_overlap: bool = False, **kwargs): ...

Then bibentry_detection_predictor's interface calls doc.annotate(bib_entries=..., allow_overlap=True).

Option B — de-overlap in the predictor before annotating (no mmda core change): merge or drop the lower-confidence of any two overlapping predicted span groups before doc.annotate:

def _resolve_overlaps(span_groups):
    span_groups = sorted(span_groups, key=lambda sg: (sg.start, -sg.end))
    kept, last_end = [], -1
    for sg in span_groups:
        if sg.start >= last_end:        # disjoint -> keep
            kept.append(sg); last_end = sg.end
        # else: overlaps a kept group -> drop or merge (by score)
    return kept

Option A is the durable fix (it removes the failure for all three services and any future overlap-producing predictor); Option B is a per-service mitigation if a core change is undesirable.

---

Filed from production failure analysis of the TIMO invocations table. Related: #206 (older catch-all failure log), #250 (page-index IndexError family).