Judging criteria look simple on paper. You list what matters, assign some weightings, and let judges do their work, however, in practice, most criteria fail the people who need to use them.
Judges receive vague prompts like “evaluate innovation” or “assess impact” without clarity on what innovation means in this context, or how to distinguish significant impact from modest improvement. They’re left to interpret, and different judges interpret differently.
Scores vary not because judges disagree about quality, but because they’re answering different questions. This isn’t judge failure. It’s criteria failure.
Well-designed criteria tell judges exactly what to look for, how to distinguish levels of quality, and where to focus their limited attention. They turn subjective assessment into structured evaluation, and they make scores defensible because everyone applied the same standards.
This guide covers how to write judging criteria that judges can actually use, criteria that produce consistent results, reduce cognitive load, and create outcomes you can explain and defend.
What judges need
There’s often a gap between what program organisers provide and what judges require to do their job well.
What organisers typically provide
- A list of evaluation areas (innovation, impact, execution, etc.)
- Percentage weightings for each area
- A scoring scale (1–5 or 1–10)
- A deadline
What judges actually need
- Clear definitions of what each criterion means in this specific context
- Concrete indicators that distinguish good from excellent
- Guidance on what evidence to look for
- Calibration on how to apply the scale consistently
- Realistic time expectations for thorough evaluation
The gap between these lists explains why judging often feels inconsistent. Organisers assume the criteria are self-explanatory, but judges are left wondering what “innovation” actually looks like for this category, or whether a score of 4 means “good” or “almost excellent.”
The judge experience matters
Judges are typically busy professionals volunteering their time and expertise. When criteria are unclear, they face a choice: spend extra time figuring out what you meant, or make quick interpretations and move on. Most choose the latter, because they have other demands on their time.
Vague criteria don’t just produce inconsistent results. They frustrate judges, reduce engagement, and make recruitment harder for future cycles. Judges who feel set up to fail rarely return.
The most common criteria problems
Before writing better criteria, it helps to understand how they typically go wrong.
The vague abstraction
Criteria like “demonstrates excellence” or “shows leadership” sound meaningful but provide no guidance. What does excellence look like? Leadership in what sense? Judges fill in their own definitions, and those definitions vary.
The kitchen sink
Criteria that try to capture everything become impossible to apply. “Evaluate the innovation, market impact, technical execution, team capability, sustainability, and future potential” in a single criterion overwhelms judges and produces inconsistent weighting of sub-elements.
The false precision
Detailed rubrics that describe ten levels of performance for each criterion create an illusion of objectivity. In practice, judges can’t reliably distinguish between a 6 and a 7 on a ten-point scale when differences are described in subtle language variations.
The obvious checkbox
Criteria that any reasonable entry would satisfy don’t discriminate. “Entry is relevant to the category” or “submission is complete” aren’t evaluation criteria, they’re eligibility requirements.
The unmeasurable aspiration
Criteria requiring information judges can’t access from submissions create frustration. “Long-term industry impact” can’t be evaluated if judges only see a one-year project summary.
The insider assumption
Criteria using jargon or assuming context that judges may not share. External judges especially may not know what “demonstrates alignment with industry best practice” means without definition.
How to write criteria judges can actually apply
Effective criteria share several characteristics. They’re specific, observable, and graduated , meaning judges can see the difference between levels of quality.
Start with the question, not the label
Instead of starting with a criterion label like “Innovation,” start with the question you want judges to answer:
- “How novel is this approach compared to existing alternatives?”
- “Did this work introduce something genuinely new, or improve something existing?”
- “Would practitioners consider this a meaningful advance?”
The question format forces clarity about what you’re actually asking judges to evaluate.
Define your terms explicitly
Every key term needs definition within your context. “Impact” means different things in different industries and categories.
Vague: “Evaluate the impact of this initiative.”
Clear: “Evaluate measurable outcomes achieved. Strong entries will demonstrate specific results, revenue growth, efficiency gains, customer improvements, or other quantifiable changes, with evidence supporting the claimed outcomes.”
Describe what good looks like
| Score | Description |
|---|---|
| 5 | Exceptional: among the best examples judges have encountered; sets new standards for the category |
| 4 | Strong: clearly above typical quality; demonstrates excellence with minor limitations |
| 3 | Competent: meets expectations for the category; solid but not distinctive |
| 2 | Developing: shows potential but has significant gaps or weaknesses |
| 1 | Insufficient: does not meet basic expectations for the category |
These descriptions aren’t perfect, but they give judges shared reference points. A 5 isn’t just “better than 4”, it’s genuinely exceptional.
Make criteria observable
Judges can only evaluate what they can see in submissions. Criteria should reference observable evidence rather than internal states or unmeasurable qualities.
Difficult to observe: “The team was passionate about this project.”
Observable: “The submission demonstrates deep engagement with the problem, including specific details about challenges encountered and how they were addressed.”
Limit the number of criteria
More criteria doesn’t mean better evaluation. It means diluted attention and judge fatigue.
For most categories, 4–6 criteria provide sufficient coverage without overwhelming judges. Each criterion should represent a meaningfully distinct dimension of quality.
If you have ten criteria, some probably overlap or could be consolidated. If every criterion is weighted at 10%, none of them matter much individually.
Weight criteria to reflect priorities
Not all criteria matter equally. Weightings should reflect what genuinely distinguishes excellence in this category. If innovation is the core purpose of the award, weight it heavily perhaps 40%. If execution quality matters but isn’t the primary focus, weight it lower, perhaps 15%.
Weightings also communicate to entrants. High weightings signal where to focus their submissions.
Examples: vague criteria versus usable criteria
Example 1: Innovation
Vague:
Evaluate the innovation demonstrated in this entry.
Usable:
Innovation (30%)
How novel is the approach, solution, or methodology? Consider:
- Does this represent a genuine advance over existing alternatives, or incremental improvement?
- Would industry practitioners consider this approach meaningfully new?
- Has the entry clearly articulated what makes this innovative and why it matters?
Score 5: Genuinely pioneering, introduces concepts or approaches not previously seen in this context
Score 3: Notable improvement, advances existing approaches in meaningful ways
Score 1: Conventional, follows established patterns without meaningful innovation
Example 2: Impact
Vague:
Assess the impact of this work.
Usable:
Results and impact (35%)
What measurable outcomes did this work achieve? Consider:
- Are specific results provided with supporting evidence?
- Are the claimed outcomes proportionate to the investment and effort described?
- Is there evidence beyond self-reported metrics (customer testimonials, third-party data, etc.)?
Score 5: Exceptional outcomes with strong evidence, results significantly exceed typical achievements in this category
Score 3: Solid outcomes with adequate evidence, results meet expectations with reasonable documentation
Score 1: Limited or unsupported outcomes, results are vague, minor, or lack credible evidence
Example 3: Execution
Vague:
Rate the quality of execution.
Usable:
Execution quality (20%)
How well was this work planned and delivered? Consider:
- Was the approach appropriate for the challenge described?
- Were obstacles identified and addressed effectively?
- Does the submission demonstrate thoughtful decision-making throughout the process?
Score 5: Exemplary execution, demonstrates sophisticated planning, adaptation, and delivery
Score 3: Competent execution, shows reasonable planning and delivery with some limitations
Score 1: Poor execution, significant gaps in planning, delivery, or decision-making
Example 4: Presentation quality
Vague:
Consider how well the entry is presented.
Usable:
Clarity and evidence (15%)
How clearly does the submission communicate its case? Consider:
- Is the entry well-structured and easy to follow?
- Are claims supported with specific evidence rather than general assertions?
- Does the submission address the judging criteria directly?
Note: Strong writing alone should not compensate for weak substance. This criterion evaluates whether the case is made clearly, not whether the prose is polished.
Aligning criteria with entry forms
Criteria and entry forms should work together, and when they don’t, judges struggle to find the information they need, and entrants provide content that doesn’t address what judges evaluate.
The alignment principle
| Criterion | Corresponding entry question |
|---|---|
| Innovation (30%) | “What is novel about your approach? How does it differ from existing alternatives?” |
| Results and impact (35%) | “What measurable outcomes did you achieve? Provide specific metrics and evidence.“ |
| Execution quality (20%) | “Describe how you planned and delivered this work. What challenges did you encounter and how did you address them?” |
| Clarity and evidence (15%) | (Assessed from overall submission quality) |
When alignment breaks down
Misalignment creates problems for everyone:
- Entrants provide information judges don’t need while missing what they do
- Judges hunt through submissions for relevant content
- Scores vary based on how well judges extract information from poorly structured submissions
If judges consistently comment that entries don’t give them what they need, the problem is usually criteria-to-entry misalignment rather than entrant failure.
Testing your criteria before judging begins
Criteria that seem clear to their authors often confuse the people who must apply them. Testing reveals problems before they affect real evaluation.
The colleague test
Give your criteria to someone unfamiliar with the program and ask them to explain what each criterion means. If they can’t, or if their explanation differs from your intent, the criteria need clarification.
The sample entry test
Apply your criteria to a past entry or hypothetical submission. Can you score it confidently? Do you find yourself unsure which criterion covers certain content? Do the criteria capture what actually matters about the entry?
The calibration exercise
Have two or three people independently score the same sample entry, then compare results. Where scores diverge significantly, discuss why. The conversation usually reveals criteria that need clearer definition.
The judge feedback loop
After judging, ask judges what was unclear or difficult. Their feedback identifies criteria problems for next cycle. Common feedback includes:
- “I wasn’t sure what you meant by…”
- “The entries didn’t give me information about…”
- “I found it hard to distinguish between 3 and 4 on…”
Collecting this feedback systematically improves criteria over time.
The calibration session: aligning judges before they begin
Even well-written criteria benefit from live calibration. A brief session where judges discuss sample entries together creates shared understanding that written documents can’t fully achieve.
What calibration accomplishes
- Surfaces different interpretations before they affect scoring
- Establishes consensus on what different score levels look like
- Gives judges confidence that they understand expectations
- Identifies criteria ambiguities that need clarification
Simple calibration format
- Distribute one or two sample entries in advance
- Judges score independently before the session
- Compare scores and discuss significant variations
- Identify where interpretations differed and clarify
- Re-score after discussion to confirm alignment
Even a 30-minute calibration session significantly improves scoring consistency. The investment is small compared to the credibility cost of wildly varying scores.
When criteria aren’t the problem
Sometimes scoring inconsistency comes from issues that better criteria can’t fix.
Judge qualification mismatch
Judges without relevant expertise may score confidently but inaccurately. Technical categories need technical judges. Creative categories need judges who understand the craft.
Insufficient time
Rushed judges skim entries and score quickly. Even perfect criteria can’t compensate for inadequate evaluation time. Be realistic about how long thorough evaluation requires, and don’t overload judges with more entries than they can assess properly.
Entry quality variation
When entries vary dramatically in format, length, and completeness, judges are comparing submissions that aren’t genuinely comparable. Entry design affects judging consistency as much as criteria design.
Conflicts of interest
Judges with undisclosed relationships to entrants may score strategically rather than objectively. Conflict management matters alongside criteria quality.
Checklist
Judging criteria that work
Clarity
[ ]Each criterion answers a specific question, not just a label
[ ]Key terms are defined within the program context
[ ]Criteria reference observable evidence, not internal states
[ ]Score levels are described with distinguishing characteristics
Structure
[ ]4–6 criteria maximum for most categories
[ ]Weightings reflect genuine priorities
[ ]Criteria don’t overlap significantly
[ ]Each criterion represents a distinct dimension of quality
Alignment
[ ]Entry questions correspond to judging criteria
[ ]Judges can find relevant information without hunting
[ ]Entrants understand what judges will evaluate
Testing
[ ]Criteria tested with sample entries before use
[ ]Calibration session planned or conducted
[ ]Feedback mechanism exists for judge input
[ ]Criteria reviewed and refined annually
Conclusion
Most judging criteria fail the people who need to use them. Judges receive vague prompts like “evaluate innovation” without clarity on what innovation means in this context, and scores vary not because judges disagree about quality but because they’re answering different questions. This is criteria failure, not judge failure. Avoid the common problems: vague abstractions that sound meaningful but provide no guidance (“demonstrates excellence”), kitchen-sink criteria that try to capture everything in one item, false precision with ten-point scales where judges can’t reliably distinguish adjacent levels, obvious checkboxes that any reasonable entry would satisfy, unmeasurable aspirations requiring information judges can’t access, and insider assumptions using jargon external judges don’t share.
Instead, start with the question you want judges to answer rather than a label, define every key term within your specific context, describe what good looks like at each score level with distinguishing characteristics, make criteria observable by referencing evidence rather than internal states, and limit yourself to 4–6 criteria maximum because more means diluted attention and judge fatigue.
Align your criteria with entry forms so judges can find the information they need without hunting and entrants provide content that addresses what judges evaluate, every criterion should have a corresponding entry question, and every entry question should relate to a criterion. Test criteria before judging begins using the colleague test (can someone unfamiliar explain what each criterion means?), the sample entry test (can you score confidently?), and the calibration exercise (do independent scorers converge?). Even well-written criteria benefit from live calibration where judges discuss sample entries together, because a 30-minute session creates shared understanding that written documents alone can’t achieve.
Collect judge feedback systematically after each cycle , “I wasn’t sure what you meant by…” and “I found it hard to distinguish between 3 and 4 on…” ,and use it to improve criteria over time. When criteria are clear, judges can focus on evaluating quality rather than interpreting instructions, and outcomes become defensible because everyone applied the same standards.
→ Scoring Models Explained: Numeric, Weighted, Ranked, and Hybrid : Once your criteria are clear, choose the right scoring approach. This guide covers scale design, weighting decisions, and when to use different models.
→ How to Calibrate Judges for Consistent Scoring : Even well-written criteria benefit from calibration. This guide covers how to align judges before evaluation begins.
→ How to Recruit and Brief Award Judges Effectively : Criteria are only part of judge preparation. This guide covers the full journey from recruitment through briefing.
→ Judging & Scoring Best Practices : The complete guide to designing judging processes that produce credible, defensible outcomes.
Frequently Asked Questions
Vague criteria such as “evaluate innovation” or “assess impact” leave too much room for interpretation. Without clear definitions, judges apply their own understanding of what those terms mean, resulting in scores that reflect different standards rather than different opinions on quality. When judges are effectively answering different questions, variation in scoring increases. Clear, specific criteria reduce this problem by ensuring all judges evaluate the same elements using shared definitions and expectations.
For most categories, 4–6 criteria provide sufficient coverage without overwhelming judges. Fewer criteria help focus attention on what truly distinguishes excellence and reduce cognitive load during evaluation.
Too many criteria dilute focus, create overlap, and increase fatigue. If every criterion carries a small, equal weighting, none of them meaningfully shapes outcomes, and judges may struggle to prioritise what matters most.
A usable criterion answers a specific question, defines key terms clearly, and points judges toward observable evidence. Instead of broad labels like “excellence,” it specifies what judges should look for and how to distinguish levels of quality.
Strong criteria:
- Define terms within the program’s context
- Describe what different score levels look like
- Reference evidence visible in submissions
- Avoid combining multiple dimensions into one vague prompt
This structure turns subjective impressions into structured evaluation.
Score levels should include clear, distinguishing descriptions rather than subtle wording differences. For example, defining a score of 5 as “exceptional, among the best examples encountered” and a 3 as “competent, meets expectations but not distinctive” provides shared reference points.
Overly detailed ten-point scales with minor linguistic variations create false precision. Judges generally cannot reliably differentiate between closely worded adjacent scores. Clear distinctions between fewer levels improve reliability.
Judges can only evaluate what entrants provide. When criteria and entry questions are misaligned, judges must search for relevant information or guess based on incomplete answers, leading to inconsistent scoring.
Strong alignment ensures:
- Each criterion corresponds to a specific entry question
- Entrants know exactly what evidence to provide
- Judges can easily find and assess relevant information
When alignment works, evaluation becomes faster, fairer, and more defensible
A calibration session is a structured discussion where judges independently score one or two sample entries and then compare results before formal judging begins. Differences in scoring are discussed to clarify interpretations and align expectations.
Even a short calibration session can:
- Surface hidden disagreements about criteria meaning
- Establish shared standards for score levels
- Increase judge confidence and consistency
This small time investment significantly improves scoring reliability.
Criteria should be tested before judging begins to identify ambiguity or overlap. Practical methods include:
- Colleague test: Ask someone unfamiliar with the program to explain each criterion.
- Sample entry test: Apply the criteria to a past or mock submission to see if scoring feels clear and confident.
- Small-group scoring: Have multiple people score the same sample and compare differences.
Testing reveals gaps and confusion early, when adjustments are still easy to make.
The Awards Platform allows organisers to configure entry forms and judging criteria together, ensuring alignment between what entrants are asked and what judges score. Entry forms can include tailored questions, image uploads, and external links, mirroring each weighted criterion.
On the judging side, criteria descriptions appear directly within the scoring interface, so judges see guidance while evaluating. The platform applies whatever structure is designed, meaning consistent results depend on the clarity and specificity of the criteria created by organisers.