Evaluation & AI Reliability
Jordan Lee
Jordan teaches how to know whether an LLM feature is good enough to ship — and how to notice when it stops being good enough.
Sample profileLoading page
Evaluation
LLM Evaluation & Reliability
Measure what your LLM system does and catch regressions before users do.
- Level
- Intermediate
- Format
- Online · Self-paced
- Duration
- 6 weeks (sample schedule)
- Effort
- 4–6 hrs/week (suggested)
- Modules
- 5 modules
- Price
- Price on request (demo)
- Availability
- Open enrollment (demo)
Register interestEditable sample values
01
Overview
LLM outputs vary, and a prompt change that fixes one case can quietly break three others. Evaluation is how you replace guesswork with evidence. This program teaches you to build evaluation into your development workflow from the start.
You will create test sets that reflect real usage, write deterministic checks where they are possible, and use model-graded evaluation where they are not — while learning where LLM-as-judge is unreliable and how to validate it against human labels.
The program ends with a working evaluation bench: a versioned dataset, a set of scorers, and a pytest-based regression suite that runs in CI and reports how each change affects quality, latency, and cost.
fig. — eval pass rate by prompt versionIllustrative
- groundedness
- format validity
02
Prerequisites
- Comfortable writing Python and tests with pytest
- Experience building at least one small LLM-powered feature
- Basic statistics: averages, variance, and sample size
03
Curriculum
Module 01: Why LLM Evaluation Is Different
Understand the sources of variability in LLM systems and what a useful evaluation looks like.
- 01.1Non-determinism, sampling, and output variance
- 01.2Defining quality criteria for a specific task
- 01.3Offline evaluation versus online monitoring
Module 02: Building Test Sets
Create datasets that cover the inputs your system will actually see.
- 02.1Sourcing examples from logs, domain experts, and edge cases
- 02.2Generating synthetic test cases and reviewing them
- 02.3Labeling guidelines and reference answers
- 02.4Versioning datasets alongside code
Module 03: Scoring Methods
Choose the right check for each criterion, from exact matching to model-graded rubrics.
- 03.1Deterministic checks: schema validation, exact match, and assertions
- 03.2Groundedness and citation checks for RAG outputs
- 03.3LLM-as-judge: rubric design, position bias, and self-preference
- 03.4Validating automated judges against human labels
Module 04: Failure Analysis
Turn a list of failing cases into a prioritized set of fixes.
- 04.1Reading outputs systematically and tagging error types
- 04.2Separating retrieval, prompting, and model failures
- 04.3Prioritizing fixes by frequency and impact
- 04.4Adding each fixed failure to the test set
Module 05: Regression Testing in CI
Make evaluation a routine part of shipping changes.
- 05.1Structuring a pytest-based evaluation suite
- 05.2Thresholds, tolerances, and handling flaky results
- 05.3Tracking quality, latency, and cost per run
- 05.4Reporting results on pull requests
04
Skills and project outcomes
Skills you’ll practice
- Test set design and curation
- Deterministic and schema-based checks
- Groundedness and citation checks
- LLM-as-judge rubric design and validation
- Error categorization and failure analysis
- Prompt and pipeline regression testing
- Evaluation in CI pipelines
What you’ll build
- A versioned test set with labeling guidelines
- A library of scorers combining deterministic and model-graded checks
- A pytest regression suite that runs in CI
- A failure analysis report with categorized errors and recommended fixes
Related project briefs
Outcomes describe skills and artifacts you will practice. No certification, employment or career outcome is guaranteed.
05
Mentors
AI Agents & Workflows
Maya Chen
Maya teaches agent design as workflow engineering: well-scoped tools, explicit state, clear stopping rules and a plan for every way a step can fail.
Sample profile06
Program FAQs
Sometimes, within limits. Model-graded checks can be useful for criteria that are hard to test deterministically, but they have known biases. The program shows how to measure a judge's agreement with human labels before relying on it.
07
Where this fits
Interested in LLM Evaluation?
Register your interest and tell us about your experience. We’ll suggest where to start.