Abschnitt 4 · Verifikation

Qualität, Sicherheit und Betrieb evaluieren

Eine eindrucksvolle Demo beweist keine Zuverlässigkeit. Nutze repräsentative Fälle, Baseline, Fehleranalyse und klare Schwellen.

Mittelstufe25 Minuten

Danach kannst du

Build an evaluation set tied to real users and failure costs.
Measure quality separately from safety, latency and cost.
Plan limited rollout, monitoring, fallback and rollback.

Fünf Teile einer glaubwürdigen Evaluation

Task and baseline

Define observable success and compare against the current process or a simpler method. Include typical, rare, ambiguous and adversarial cases.

Quality and calibration

Measure task utility, factual support, citation validity, consistency and appropriate abstention. A single average can hide severe failures.

Security and privacy

Test direct and indirect prompt injection, malicious retrieved content, tool abuse, data disclosure and privilege boundaries. Treat all model output as untrusted input.

Operational measures

Track end-to-end latency, tail latency, token and tool cost, availability and recovery. Version models, prompts, tools, policies, data and evals together.

Rollout and governance

Start with limited permissions and users, observe real outcomes, keep a deterministic or human fallback, and define circuit breakers and rollback before promotion.

The evidence loop

01
Define risk
02
Build representative set
03
Run baseline and candidate
04
Review failures
05
Gate and monitor

Separate construction from evaluation when possible. For variable outputs, repeat tests and examine distributions and worst cases—not only the best example.

Häufige Verwechslungen

An LLM judge can scale review, but it must be calibrated against blinded human judgments for the target task.
Safety refusal rate alone is incomplete: measure both harmful compliance and excessive refusal of legitimate requests.
Offline success does not prove production success when users, data, tools or model versions differ.

Practice: create a 12-case eval

Choose one LLM feature and build a small predeployment gate.

Include normal, edge, ambiguous, multilingual, injection and tool-failure cases.
Score factual support, task completion, appropriate refusal, latency and cost separately.
Set a blocking safety threshold plus a rollback trigger before seeing candidate results.

Quick check

Why compare with a baseline?

To learn whether added model complexity materially improves the current or simpler process under the same criteria.

Why test prompt injection if the model has no database write access?

Injection can still disclose data, corrupt answers or abuse any available read tools; least privilege limits but does not erase risk.

What must be versioned for reproducible evaluation?

At minimum the model, parameters, prompts, tool contracts, policies, evaluation data and scoring method.

Schlüsselbegriffe

evaluation
baseline
benchmark
calibration
abstention
prompt injection
least privilege
latency
observability
rollback

Frameworks und Evaluationsleitfäden

HELMHolistic framework for transparent multi-metric language-model evaluation.NIST Generative AI ProfileCross-sector risk-management profile for generative AI.OWASP Top 10 for LLM applicationsCommunity risk catalogue for common LLM application vulnerabilities.OpenAI evaluation best practicesCurrent guidance on eval design, criteria, datasets and continuous evaluation.

Fünf Teile einer glaubwürdigen Evaluation

Task and baseline

Define observable success and compare against the current process or a simpler method. Include typical, rare, ambiguous and adversarial cases.

Quality and calibration

Measure task utility, factual support, citation validity, consistency and appropriate abstention. A single average can hide severe failures.

Security and privacy

Test direct and indirect prompt injection, malicious retrieved content, tool abuse, data disclosure and privilege boundaries. Treat all model output as untrusted input.

Operational measures

Track end-to-end latency, tail latency, token and tool cost, availability and recovery. Version models, prompts, tools, policies, data and evals together.

Rollout and governance

Start with limited permissions and users, observe real outcomes, keep a deterministic or human fallback, and define circuit breakers and rollback before promotion.

Häufige Verwechslungen

An LLM judge can scale review, but it must be calibrated against blinded human judgments for the target task.

Safety refusal rate alone is incomplete: measure both harmful compliance and excessive refusal of legitimate requests.

Offline success does not prove production success when users, data, tools or model versions differ.

Practice: create a 12-case eval

Choose one LLM feature and build a small predeployment gate.

Include normal, edge, ambiguous, multilingual, injection and tool-failure cases.

Score factual support, task completion, appropriate refusal, latency and cost separately.

Set a blocking safety threshold plus a rollback trigger before seeing candidate results.

Quick check

Why compare with a baseline?

To learn whether added model complexity materially improves the current or simpler process under the same criteria.

Why test prompt injection if the model has no database write access?

Injection can still disclose data, corrupt answers or abuse any available read tools; least privilege limits but does not erase risk.

What must be versioned for reproducible evaluation?

At minimum the model, parameters, prompts, tool contracts, policies, evaluation data and scoring method.