# Agent quality engineering

The portfolio agent doubles as a reusable AI quality system. It exercises real
retrieval, tools, schema validation and conversation state in workerd. Only
external providers are substituted in CI. The same harness accepts live HTTP
transport for owner-triggered experiments; no company-specific application is
involved.

## Investigate, reproduce, compare

1. Find the failed operation in Braintrust: retrieval, answer/assessment model,
   tool, validation, follow-ups or grader. Child spans record actual operation
   boundaries and allowlisted failure categories. They exclude conversation
   text, source passages, tool arguments and visitor identifiers.
2. Preserve a curated example in `worker/evals/scenarios.json`, or extend the
   reproducible templates in `worker/evals/datasets.py`. Never copy a visitor
   conversation or employer implementation into a public regression case.
3. Run the desktop/mobile E2Es. `worker/evals/harness.mjs` carries actual answers
   forward and selects subsequent simulated questions using returned evidence.
4. Dispatch **Compare agent candidates** in GitHub Actions on main. Change one
   axis: assessment model, embedding cache, complexity routing, or conversation
   retrieval. Download the private report and inspect failures before promotion.
5. Open `/engineering/quality/`, import the report locally and independently
   label the final answer using the supplied conversation context and evidence.
   Export reviewed labels, then run the calibration command below.
6. Adopt a candidate only through a reviewed code change. Record the experiment,
   human review and previous deployable revision. CI never promotes a model.

## Experiments

`npm run quality --prefix worker` uses the authenticated evaluation endpoint.
Supply `RAG_EVAL_SECRET` through the environment; never put it in a command or
report. Targets are restricted to the reviewed Worker hostname family.

| Setting                                               | Default / purpose                                                         |
| ----------------------------------------------------- | ------------------------------------------------------------------------- |
| `QUALITY_EXPERIMENT_KIND`                             | `model`; also `embedding_cache`, `adaptive_routing`, `simulation`         |
| `QUALITY_CASE_IDS`                                    | Two reviewed assessment cases, or three generated simulation cases        |
| `QUALITY_REPEATS`                                     | 3; alternate baseline/candidate order each repeat                         |
| `QUALITY_COST_LIMIT`                                  | $0.10 in reported LLM costs; optional grader cost included                |
| `QUALITY_JUDGE`                                       | `false`; optional extra semantic grading, not human-calibrated            |
| `QUALITY_ENFORCE_GATE`                                | Set `true` to fail on blocked/inconclusive results; Actions defaults true |
| `QUALITY_PUBLISH`                                     | Set `true` for a sanitized public report; Actions defaults true           |
| `QUALITY_BASELINE_COMMIT`, `QUALITY_CANDIDATE_COMMIT` | Expected deployment revisions; Actions defaults current SHA               |
| `QUALITY_BASELINE_URL`, `QUALITY_CANDIDATE_URL`       | Approved deployment/preview endpoints                                     |

The simulation experiment compares latest-question retrieval with conversation
context retrieval using the same chat model. Assessment experiments change only
the chosen model or optimization flag. The cache stores bounded, short-lived
embeddings keyed by a hash of model and query. Routing tries the alternate structured model for short assessments; longer
requests retain the baseline model. Cost savings and reliability are measured,
not assumed.
These two optimizations are authenticated evaluation options, disabled for
public traffic pending a reviewed experiment.

The cost guard stops **between requests**, including between conversation turns.
One request can exceed the remaining guard. Missing usage/cost stops further
calls. Reported LLM costs exclude embeddings and infrastructure; the provider's
account spending limit remains the hard cap. No retry or judge runs by default.

## Gate and interpretation

The server recalculates the gate from sanitized rows, discarding supplied
conversations and claimed gate results. The planned case list prevents budget
cutoffs from hiding unexecuted scenarios. Eligibility requires complete pairs,
matching corpus versions, consistent code/prompt revisions within each variant,
at least three repeats over two scenarios, fully reported costs, all candidate
workflow checks passing, no paired workflow regressions, P95 latency within
1.25 times baseline, and total reported cost within 1.1 times baseline.

Insufficient/incomplete/version-mismatched samples are **inconclusive**. Complete
samples with regressions are **blocked**. A one-pair smoke run demonstrates the
pipeline but cannot justify promotion. Intervals are descriptive Wilson 95%
intervals; repeats are correlated, cases are curated and workflow passes are not
semantic accuracy. Aggregate records retain actual model, prompt, corpus, dataset
and code versions so old experiments remain inspectable after deployment.

## Human review and grader calibration

The annotation workspace reads files into browser memory and downloads labels.
It uploads neither reports nor labels, disables page analytics, and records no visitor conversations.
Context and retrieved passages are shown as escaped text. Review the final
answer independently before revealing the automated verdict. Record reviewer,
time, rationale and development/held-out split. Set aside held-out cases before
rubric development and do not tune against their results.

```sh
python3 worker/evals/datasets.py generate --output conversations.json
python3 worker/evals/datasets.py calibrate \
  --report report.json --labels agent-human-labels.json --output calibration.json
```

Labels bind to the exact report SHA-256 and rubric version. Duplicate labels,
invalid splits or mismatched reports are rejected. The tool measures final-answer
grader agreement and false-pass rates separately for development and held-out
cases. It never fabricates labels or certifies a grader as calibrated. Human
review must assess sample coverage and uncertainty; empty splits are unknown.
Synthetic CI reviewers and model outputs test this workflow and are never
published as human calibration evidence.

## Release and rollback

**Validate and deploy sites** gates main with browser workflows, real-Worker
integration checks, both actual site outputs, Worker typechecking and a dry-run
bundle. Public experiment summaries are separate from deployment status: the
manual paid job does not alter production settings. Private synthetic artifacts
are retained for 30 days in GitHub; public aggregates contain no answers.

After adoption, compare deployed trace versions with the approved experiment.
If outcomes regress, revert the adoption commit on main and rerun the same CI
and deployment pipeline. Corpus and evidence snapshots are versioned separately;
check their hashes as well as code SHA. Existing evidence retirement checks must
remain intact when reverting so withdrawn employer material cannot reappear.