Experiments and holdouts

Experiments estimate whether delivered clarification reduces downstream failure. Sessions—not requests—are analysis unit.

#Arms

• GATED → eligible questions can be returned.
• HOLDOUT → request is scored and recorded, but questions are suppressed.

Assignment hashes session_id:salt with SHA-256 and compares first 64 bits with configured gate fraction. Same session stays in same arm. When experiment is disabled, every session is GATED.

#Outcome

Each arm counts unique gated sessions and unique sessions with at least one failure. Absolute reduction is:

holdout_failure_rate − gated_failure_rate

Confidence interval is unpooled two-proportion 95% z-interval:

SE = sqrt(p_holdout(1-p_holdout)/n_holdout + p_gated(1-p_gated)/n_gated)
CI = reduction ± 1.96 × SE

Relative reduction and bounds divide absolute values by holdout rate. They are undefined when holdout rate is zero.

#Minimum samples

API sets target at 200 sessions per arm and returns minimum_sample_warning whenever either arm has fewer than 200: “Directional only: fewer than 200 sessions in at least one arm.”

Confidence interval represents sampling uncertainty under model assumptions. It does not correct signal-label error, interference, traffic shifts, or repeat-user dependence.

#Configure

PUT /v1/config accepts:

{
  "experiment": {
    "enabled": true,
    "gate_fraction": 0.5
  }
}