AI quality engineering
The portfolio agent is a working system for investigating retrieval, tool use, multi-turn behavior and structured answers. This workspace makes the evaluation process inspectable: diagnose a failure, preserve it as a regression case, compare a candidate and make a release decision.
Trace the operation that failed
Braintrust records retrieval, individual model calls, tool execution and assessment validation as child spans. Spans include timings, usage and fixed failure categories. Production traces exclude conversations, answers, retrieved text, tool arguments and visitor identifiers. Evaluation artifacts contain curated test conversations for review.
Run reproducible conversations
A shared harness runs against the real Worker in browser integration tests and against live model APIs in owner-triggered experiments. Simulated users follow the agent’s actual answer and sources, then introduce corrections. Dataset versions and development/held-out provenance travel with the cases.
Inspect the reusable harness · Inspect the tracing implementation
Read the experiment and rollback runbook · Inspect the Actions workflow
Baseline and candidate experiments
Loading measured experiments…
Review and calibrate the grader
Import an owner-generated evaluation artifact to review answers. The file stays in this browser; labels download locally. Human review is separate from automated workflow checks.
Release criteria and optimization
PRs must pass the browser and real-Worker workflow checks. Paid experiments run on demand with a reported-cost guard. An automated candidate gate requires complete pairs, matching corpus versions, at least three repetitions across two scenarios, complete cost reporting, no workflow regressions and bounded latency/cost increases. Semantic quality and grader calibration require human review. A failed or inconclusive experiment preserves the deployed model.
Embedding caching and routing by assessment complexity are evaluation options. Experiments can isolate one change while keeping the other settings fixed. Production adoption requires a reviewed change with recorded evidence and a rollback revision.