Skip to content
CodeWiki
Practice
Paths
Tracks
Cheatsheets
Playground
Glossary
AI era
Search
⌘K
English
English
Chinese
Practice
Paths
Tracks
Cheatsheets
Playground
Glossary
AI era
English
English
Chinese
Practice
/
Quiz
/
Coding in the AI era
/
LLM application evals
Before an LLM judge blocks releases, what is the most useful calibration step?
from LLM application evals
Node 24
advanced
3 min
Before an LLM judge blocks releases, what is the most useful calibration step?
Compare its criterion-level verdicts with blinded human labels on clear and boundary cases, then inspect false passes and false failures
Raise the judge temperature until its explanations become more varied
Give the judge the candidate model name so it can account for model quality
Accept high agreement between two copies of the same judge prompt as ground truth
Check
Ask AI about this kata
Report an error
previous kata
A candidate delta is -1 percentage point, the allowed regression is -2 points, and the co…
Quiz
next kata
Which evaluation-set design gives the strongest release evidence for a customer-support a…
Quiz
A new version is available
Reload