Definition · foundations
LLM as judge
LLM as judge is the use of a language model to grade, rank, or critique outputs under a stated rubric. You can spot LLM as judge in practice when the record shows what the system observes, what choice it makes, and what changes next, without asking the reader to guess or trust a slogan.
157 definitions656 sourced recordsNamed source: ACL Anthology
Why does LLM as judge matter?
Model judging can cover many outputs quickly, but you still need calibration, order controls, and human checks for disputed cases. For LLM as judge, start with three checks: What can the system observe? What choice can it make? What changes after it acts? The answers should point to visible evidence, not a promise.
What does LLM as judge look like?
A language model scores summaries against a rubric after its scores are compared with a labeled reviewer set. This example makes LLM as judge visible by naming the actor, the action, and the result in practice. It is an illustration for readers, not a claim that every product behaves this way.
What is LLM as judge easy to confuse with?
LLM as judge is the evaluation method. A judge model is the model assigned to perform it. The closest entries here are least privilege, LLM-profiled role, judge model. Compare their definitions with LLM as judge before using the names as if they mean the same thing.
How certain is this definition?
This definition is provisional. ACL Anthology supports the version you see today, but a better or more direct source may change it. Open the cited page and check its date before you use this entry to choose a product, write a policy, set a safety control, or design a system.
Who introduced the term LLM as judge?
We have not verified who first introduced LLM as judge. The source supports a useful definition, but it may not be the earliest use. Until a dated primary record settles the question, we leave the origin open. A popular article or product page is not enough to name the person who coined a term.
What should you understand before LLM as judge?
Read artificial intelligence (AI) first when LLM as judge depends on a more basic idea or mechanism. The link gives you a useful route through the glossary. It does not mean every author teaches the subject in this order, so follow the source when the sequence matters.
What should this term help you answer?
Use LLM as judge to ask what an agent can observe, decide, change, remember, or hand off. The questions below turn the definition into a practical check. They are prompts for your own work, not claims made by the source, so change them to fit the system you are examining.
- What can the system observe?
- What choice can it make?
- What changes after it acts?
Which source supports this definition of LLM as judge?
ACL Anthology supports this working definition of LLM as judge. We checked the link on 2026-07-12. Open the original record for its context, methods, limits, and publication details. The short explanation here is a guide to LLM as judge; the linked source remains the evidence to inspect when the wording or boundary matters.
source foundLLM as judgeChecked 2026-07-12→