Setting up CI evals

Runner fetches tenant evals, executes them against gate or agent command, prints table, optionally writes JUnit, and exits nonzero on regression.

#Local gate run

pip install -e sdk
export INTENTLAYER_BASE_URL=https://intentlayer-staging.example.com
export INTENTLAYER_API_KEY=il_...
intentlayer-evals run \
  --against gate \
  --junit artifacts/intentlayer-evals.xml

Exit codes: 0 all pass; 1 regression; 2 config, network, auth, or malformed export.

#Stable sampling

intentlayer-evals run --against gate --sample 50 --seed 2026

Evals sort by ID. Same export/sample/seed selects same IDs. Muted evals are not fetched. Never randomize seed per pull request.

#GitHub Actions

jobs:
  intentlayer-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./ci/github-action
        with:
          base-url: ${{ vars.INTENTLAYER_STAGING_URL }}
          api-key: ${{ secrets.INTENTLAYER_API_KEY }}
          junit-path: artifacts/intentlayer-evals.xml

Reference action installs bundled Python SDK and writes result table to GITHUB_STEP_SUMMARY. GitLab example exists at ci/gitlab-ci.example.yml.

#Agent command mode

intentlayer-evals run \
  --against agent \
  --cmd "python my_agent.py" \
  --junit artifacts/agent-evals.xml

Runner pipes eval input plus newline to stdin. Context is not passed in v0. Shell operators are not evaluated. Failure means nonzero exit, timeout, or known textual failure marker. It does not judge semantic answer quality.