Real-data validation

Study tests whether pre-run ambiguity score predicts later narrow misunderstanding signals in real conversation datasets.

#ABCD result

On 10,000 ABCD records, detector marked 1,086 conversations (10.9%) with at least one signal. Gate caught 1,229/1,231 detected events (99.8%) at threshold 0.55.

That recall is not selective: gate marked 76,927/76,959 evaluable requests (100.0%) high ambiguity. Request precision was 1.6%; top-decile lift 1.10×. Available evidence says heuristic catches nearly everything by flagging nearly everything.

ABCD is crowdsourced human-to-human customer-support role-play with fictional scenarios, not production traffic and not human-to-AI.

#Method

Every user turn is scored before downstream text inspection. Independent detector then looks for explicit correction, frustration, or lexical rephrase similarity at least 0.62, separated by assistant text. No detector signal is given to gate.

Reference mix is dominated by lexical similarity: 1,193 of 1,231 events. Labels are heuristic, not ground truth. Silent failures are unobservable.

#Reproduce

uv run --python 3.13 --with-requirements realdata/requirements.txt \
  python realdata/download.py
uv run --python 3.13 --with-requirements realdata/requirements.txt \
  python realdata/study.py

Default seed 20260726, up to 10,000 eligible English conversations/source, pinned revisions, threshold 0.55.

#TODO after scorer-fix merge

append before/after selectivity metrics, exact scorer commit, and migration story. Do not replace current result until scorer-fix branch is merged and study rerun.