Loading page
P.03Evaluation
Build a repeatable evaluation harness that runs test cases against prompts and models, scores the results and tracks regressions between versions.
01
Most LLM features are changed by editing a prompt, swapping a model or adjusting retrieval, and then checking a handful of examples by eye. That approach misses regressions, makes comparisons subjective and leaves no record of why a change was shipped.
In this project you build an evaluation bench that treats prompts and model configurations as versioned artifacts. You write test cases, combine deterministic checks with rubric-based grading, compare runs side by side and fail a CI job when quality drops below an agreed threshold.
02
Test cases live in versioned YAML or JSON files. A Python runner executes them against one or more configurations, applies a set of scorers and writes results to Postgres. A small dashboard compares runs and highlights regressions.
Define cases
Write test cases with inputs, expected properties and tags for the behaviors they cover.
Configure targets
Describe each prompt and model configuration as a versioned, comparable record.
Run
Execute cases concurrently with caching, retries and recorded inputs, outputs, latency and token counts.
Score
Apply deterministic checks such as schema and keyword rules, plus rubric-based model grading where needed.
Compare & gate
Diff results against a baseline run, surface regressions and fail the CI job when thresholds are not met.
03
Suggested stack. Equivalent tools are fine; document the choice in your decision record.
04
Stage 01
Pick a target feature, write 40–60 test cases covering typical, edge and adversarial inputs, and tag them by behavior.
Stage 02
Build the runner with caching and result storage so any run can be reproduced and compared later.
Stage 03
Implement deterministic scorers first, then add a rubric-based grader and check it against a sample of manual labels.
Stage 04
Add baseline comparisons, regression highlighting and a CI job that runs a fast subset on every change.
05
06
Build the foundations, projects, and habits needed to design more dependable AI applications.