A Practical Checklist for Evaluating LLM Outputs

Most LLM features ship after someone tries a dozen prompts by hand and decides the output “looks good.” That works until the first prompt tweak, model upgrade, or data change, when nobody can say whether things improved or regressed. Evaluation replaces that impression with a repeatable measurement.

This checklist is deliberately practical. It assumes you have a working feature and want a lightweight evaluation loop you can run locally and in CI, then grow over time.

Define what “good” means first

Before writing any test, write down the qualities that matter for this specific feature. “Accurate and helpful” is too vague to grade. Break quality into properties that a person, or a script, could check on a single output.

• Correctness: does the answer agree with the source or reference answer?
• Grounding: is every claim supported by the provided context, with no invented details?
• Format: does the output parse, match the schema, and respect length limits?
• Scope: does the feature decline questions it shouldn't answer?
• Safety and policy: does it avoid content your product must never produce?

Rank these. When two properties conflict — for example, completeness versus brevity — you need to know which wins before you can interpret a score.

For each property, write a pass condition concrete enough that two reviewers would usually agree. “Mentions the 30-day refund window when the question is about refunds” is gradable; “explains the refund policy well” is not. If you can't phrase a pass condition, that property probably needs human review rather than an automated check, and it's better to know that up front.

Build a representative eval set

An eval set is a versioned collection of inputs plus whatever you need to judge the output: a reference answer, required phrases, forbidden phrases, or a flag saying the feature should refuse. Store it as JSONL next to your code so changes are reviewed like any other change.

1{2  "id": "refund-window-01",3  "question": "Can I still get a refund 20 days after purchase?",4  "must_include": ["30 days"],5  "must_not_include": ["guaranteed"],6  "should_refuse": false,7  "tags": ["billing", "policy"]8}

• Pull real inputs from logs (with sensitive data removed) rather than inventing tidy examples.
• Cover the common cases, then deliberately add edge cases: ambiguous questions, out-of-scope requests, empty or very long inputs.
• Add every production bug you fix as a new case so it can't quietly return.
• Tag cases by category so you can see where quality changes, not just whether the overall number moved.

Layer checks from cheap to expensive

Deterministic checks

Start with assertions a script can evaluate exactly: the output parses as JSON, required fields exist, a required phrase appears, cited IDs are a subset of retrieved IDs, length is under a limit. These checks are fast, free, and unambiguous, and they catch a surprising share of regressions.

Model-graded checks

For properties like grounding or relevance, a second model call with a narrow rubric can act as a grader. Keep the rubric specific (“Is every factual claim in the answer supported by the passages? Answer yes or no, then list unsupported claims.”) and calibrate it: grade a sample by hand and check that the model grader agrees with you often enough to be useful.

Human review

Humans remain the reference for nuanced quality. You don't need to review everything; review a rotating sample, plus every case where automated checks disagree with each other. Record decisions so they can become new eval cases.

A minimal eval harness

The harness below uses pytest so each case becomes an individually reported test. It assumes your application exposes an answer_question function that returns the text plus retrieval metadata. Adapt the names to your code.

1import json2from pathlib import Path3 4import pytest5 6# Your application entry point. Assumed to return an object with7# .text, .refused, .cited_ids and .retrieved_ids attributes.8from app.answer import answer_question9 10CASES_PATH = Path("evals/support_cases.jsonl")11CASES = [json.loads(line) for line in CASES_PATH.read_text().splitlines() if line.strip()]12 13 14@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])15def test_support_answer(case: dict) -> None:16    result = answer_question(case["question"])17 18    if case.get("should_refuse"):19        assert result.refused, "expected a refusal for an out-of-scope question"20        return21 22    text = result.text.lower()23    for phrase in case.get("must_include", []):24        assert phrase.lower() in text, f"missing required phrase: {phrase!r}"25    for phrase in case.get("must_not_include", []):26        assert phrase.lower() not in text, f"contains forbidden phrase: {phrase!r}"27 28    # Grounding: the answer may only cite passages that were actually retrieved.29    assert set(result.cited_ids) <= set(result.retrieved_ids), "cited an unretrieved source"

Read results without fooling yourself

LLM outputs vary between runs, so a single pass rate is a noisy signal. A change from, say, 82% to 85% on a small set may be noise. Run flaky cases several times, compare per-tag results, and always read the actual failing outputs before celebrating or panicking.

• Pin the model version, temperature, and prompt version for each run, and record them alongside results.
• Diff failures between runs: which cases newly fail and which newly pass matters more than the aggregate.
• Watch for eval-set overfitting. If you tune prompts against the same cases for weeks, hold out a set you don't look at during iteration.

Track latency and token usage alongside quality in the same report. A prompt change that fixes two cases but doubles the average response length is a trade-off the team should decide on explicitly, not discover later on an invoice or a latency dashboard.

The checklist

1. Quality properties are written down and ranked for this feature.
2. The eval set is versioned, sourced from real inputs, and tagged by category.
3. Edge cases and out-of-scope requests are represented, not just happy paths.
4. Deterministic checks run on every commit.
5. Model-graded checks use a narrow rubric and have been spot-checked against human judgment.
6. Model, prompt, and configuration versions are recorded with every eval run.
7. Results are compared per case and per tag, not only as one aggregate number.
8. Every fixed production bug becomes a new eval case.