IntentBench

IntentBench is 300-item ambiguity benchmark for agent requests: 220 curated template-simulator outputs and 80 hand-written edge cases, 270 English and 30 Polish.

#Labels and dimensions

Items are clear or ambiguous. Ambiguous items list missing dimensions and one gold question. Data covers goal, output format, scope, constraints, pronouns, idioms, conflicts, destructive actions, and multi-intent requests.

All cases are synthetic or hand-written; no customer, production, scraped, or personal data. Dataset uses CC BY 4.0.

#Validate and run

python -m bench.validate
python -m bench.score --gate http://localhost:8000

INTENTBENCH_API_KEY adds bearer auth. Repeat --gate to shard across equivalent local processes.

#Metrics

• Precision/recall/F1: score at least 0.55 predicts ambiguous.
• Dimension accuracy: exact match at same threshold plus intent missing fields.
• Question proxy: fraction of gold missing dimensions covered by question terms.

Question proxy measures topical coverage, not prose quality.

#Current baseline

Measured 2026-07-26 on dataset v1:

Second row is not keyed LLM evidence.