Judges will disagree, count on it! Even with clear criteria, thorough briefing, and proper calibration, some entries will receive scores that diverge significantly across judges. One judge gives it a 5, another gives it a 3. One sees a clear winner, another sees a mediocre effort.
This isn’t necessarily a problem. In fact, some disagreement is healthy, it reflects the reality that quality assessment involves judgement, and reasonable experts can weigh factors differently. A program where all judges score identically would be suspicious, not reassuring. But not all variance is benign. Sometimes disagreement signals that something has gone wrong, unclear criteria, undisclosed conflicts, judges who aren’t engaging seriously, or fundamental misalignment about what the category is meant to recognise.
The challenge is distinguishing variance that’s normal from variance that needs intervention. Responding to every disagreement wastes time and undermines judge autonomy. Ignoring genuine problems produces results that can’t be defended. Getting this balance right is essential to credible outcomes.
This guide covers how to interpret score variance, when to investigate, when to accept disagreement, and what options you have when intervention is warranted.
Understanding why judges disagree
Before deciding whether variance is problematic, it helps to understand what causes it. Different causes suggest different responses.
Legitimate perspective differences
Judges bring different professional experiences, priorities, and reference points. A judge from a startup background may weight scrappiness and innovation differently than one from an enterprise context. A technical specialist may notice implementation details that a generalist misses,or may undervalue commercial factors that the generalist emphasises.
These differences are features, not bugs. They’re why you recruit diverse panels. When perspective differences drive variance, the solution is aggregation across perspectives, not forcing alignment.
Criteria interpretation differences
Despite calibration, judges may still interpret criteria differently. One judge reads “significant impact” as requiring quantified metrics; another accepts qualitative evidence. One interprets “innovation” as technically novel; another includes innovative application of existing approaches.
These differences are more problematic than perspective differences because they mean judges are answering different questions. If criteria interpretation is driving variance, clarification may be needed, either for current judging or for future cycles.
Scale usage differences
Some judges consistently score higher or lower than others. One judge’s 3 is another’s 4. Some avoid extreme scores; others use the full scale freely. These patterns are stable within judges but create variance across judges.
Scale usage differences are systematic, which means they can often be corrected statistically. When variance stems primarily from scale usage rather than substantive disagreement, normalisation can address it.
Entry ambiguity
Some entries are genuinely harder to evaluate than others. Submissions with mixed signals, strong on some criteria, weak on others, or with evidence that’s open to interpretation, invite disagreement in ways that clearer entries don’t.
Variance concentrated in ambiguous entries isn’t judge failure. It’s honest reflection of evaluation difficulty. The appropriate response may be accepting the uncertainty rather than forcing artificial resolution.
Quality or engagement differences
Sometimes variance reflects judge quality or engagement. A judge who’s skimming produces different scores than one reading carefully. A judge who’s disengaged or unqualified may score randomly or based on superficial factors.
This kind of variance is problematic and may require intervention, addressing the judge’s approach or removing their scores from consideration.
Undisclosed conflicts
A judge scoring an entry dramatically differently from peers might have an undisclosed relationship with the entrant, positive (inflating scores) or negative (deflating them). Variance driven by conflict is the most serious kind because it undermines integrity.
When variance suggests potential conflict, investigation is warranted before any statistical or procedural response.
When variance is normal versus concerning
Not every score difference requires action. Developing judgement about which variance matters is essential to efficient program management.
Signs that variance is likely normal
Modest magnitude. Scores that differ by one point on a five-point scale often reflect legitimate judgement variation. A 3 versus a 4 is typical; a 2 versus a 5 is more concerning.
Consistent patterns. If a judge consistently scores slightly higher or lower than peers across all entries, that’s scale usage difference, systematic and correctable, rather than entry-specific problems.
Complex or ambiguous entries. Entries with mixed signals naturally produce more variance than straightforward entries. Higher variance on difficult entries is expected.
Distributed across judges. When variance appears across multiple different judges rather than concentrated in one judge’s scores, it’s more likely to reflect entry characteristics than judge problems.
Within calibration bounds. If calibration established expected variance ranges, scores within those ranges are operating as designed.
Signs that variance may be concerning
Extreme magnitude. Scores at opposite ends of the scale, a 1 versus a 5, or a 2 versus a 5, suggest something beyond normal judgement variation.
Single judge outliers. When one judge’s scores diverge dramatically from all other judges consistently, the pattern warrants investigation.
Inconsistency within judge. A judge who scores some entries carefully and others seemingly randomly may not be engaging consistently.
Correlation with potential conflicts. Outlier scores on entries where a judge might have undisclosed relationships deserve scrutiny.
Pattern by entry characteristics. If variance systematically favours certain company types, geographies, or other entry characteristics, bias may be operating.
Late-judging patterns. Variance that increases toward the end of judging may reflect fatigue affecting scoring quality.
Investigation before intervention
When variance seems concerning, investigate before acting. Premature intervention based on incomplete understanding creates more problems than it solves.
Review the entry itself
Start by reading the entry that produced divergent scores. Sometimes you’ll immediately see why it was difficult to evaluate, mixed evidence, ambiguous fit with criteria, or factors that could reasonably be interpreted differently.
Understanding the entry helps you interpret whether the variance is entry-driven or judge-driven.
Examine judge reasoning
If judges provided written justification for scores (which they should for high-divergence situations), review what they said. Does the reasoning make sense even if the scores differ? Are judges weighing different factors, or interpreting the same factors differently?
Reasoning that’s thoughtful but different suggests perspective variance. Reasoning that’s superficial or doesn’t connect to criteria suggests engagement problems.
Look at judge patterns
Examine each diverging judge’s scores across all their entries:
- Is this judge consistently higher or lower than peers?
- Does this judge show divergence only on certain entries or across the board?
- Does the judge’s scoring suggest consistent application of some standard, even if different from peers?
Patterns help distinguish systematic tendencies from entry-specific problems.
Check for conflicts
Review whether any diverging judges might have relationships with the entrant that weren’t disclosed. Check employment histories, public associations, and any disclosed relationships that might be closer than initially apparent.
If conflict is suspected, handle it through your conflict management process rather than treating it as ordinary variance.
Consult the judge if appropriate
In some cases, reaching out to the judge for clarification is appropriate: “Your score on this entry differed significantly from other judges. Could you share a bit more about your reasoning?”
Frame this as information-gathering, not challenge. Some judges will provide reasoning that resolves concern. Others may reveal that they misunderstood something or didn’t engage deeply. The response informs next steps.
Options for handling variance
Once you understand what’s driving variance, several response options exist. Match the response to the situation.
Accept the variance and let aggregation work
For normal perspective differences and modest variance, the appropriate response is often no response at all. Multiple judges exist precisely so that aggregation smooths out individual variation.
If three judges score 3, 4, and 4, the average is 3.67 , a reasonable composite that captures the range of opinion. Trying to resolve the discrepancy would waste effort on variance that’s working as designed.
When to use this approach:
- Variance is modest in magnitude
- Judges’ reasoning is thoughtful even if different
- Entry quality is genuinely ambiguous
- Final ranking isn’t affected by the variance
Apply statistical normalisation
When variance stems from systematic scale usage differences, some judges consistently scoring higher or lower, statistical normalisation adjusts for these patterns.
Z-score normalisation converts each judge’s scores to a common scale based on their individual mean and standard deviation. A generous judge’s 4 and a harsh judge’s 3 might both convert to similar normalised scores if they represent similar positions relative to each judge’s typical scoring.
When to use this approach:
- Analysis confirms systematic judge-level patterns
- Sufficient data exists to detect patterns reliably
- You have statistical capability to implement correctly
- The adjustment is documented and defensible
Caution: Normalisation assumes that judges see comparable entry quality distributions. If one judge was assigned notably stronger or weaker entries, normalising their scores would distort rather than correct.
Request additional scoring
When variance leaves critical decisions unclear, additional evaluation can provide resolution.
- Additional judge. Assign the entry to another judge whose score can break the tie or clarify where the entry belongs.
- Re-evaluation. Ask diverging judges to reconsider with additional guidance, though this risks judges feeling pressured to change scores rather than genuinely reconsidering.
- Panel discussion. Bring diverging judges together to discuss the entry and attempt consensus. This is intensive but can produce insight that individual scoring misses.
When to use this approach:
- Variance affects important outcomes (winner determination, finalist selection)
- Other approaches haven’t resolved the uncertainty
- Resources permit additional evaluation effort
- You can implement without unduly pressuring judges
Remove outlier scores
In cases where a score appears clearly anomalous, dramatically different from all other judges with no compelling reasoning, removing it from aggregation may be appropriate.
When to use this approach:
- The outlier is extreme and unexplained
- Investigation didn’t reveal legitimate reasoning
- Conflict or engagement problems are suspected
- Keeping the score would distort outcomes
Caution: Removing scores is consequential. It should be documented, applied consistently, and used sparingly. Removing scores because they’re inconvenient, rather than because they’re genuinely problematic, undermines the independence that makes judging credible.
Adjust entries between categories
Sometimes variance reveals that an entry doesn’t fit well in its assigned category. Judges disagree because they’re each comparing the entry to different reference points.
If investigation suggests category fit is the problem, moving the entry to a better-fitting category, and having appropriate judges evaluate it there, may resolve variance more effectively than trying to force agreement in the wrong category.
When to use this approach:
- Variance appears driven by category fit questions
- A better-fitting category exists
- Logistics permit reassignment
- The entrant is informed and agrees
Use deliberation for final decisions
For finalist selection or winner determination, where variance matters most, deliberative discussion can complement independent scoring.
After independent scores are submitted, judges discuss high-variance entries together. The goal isn’t overriding independent scores but understanding whether discussion changes anyone’s view. A judge who scored low might be persuaded by factors they hadn’t considered. A judge who scored high might acknowledge weaknesses they’d underweighted.
When to use this approach:
- Decisions are high-stakes (winner determination, finalist selection)
- A small number of entries need discussion
- Judges can participate without undue time burden
- Facilitation can keep discussion productive
Documenting variance decisions
Whatever approach you take, document it. Variance handling often becomes relevant later, when results are questioned, when similar situations arise in future cycles, or when you’re evaluating whether your process is working.
What to document
The variance. Which entry, which judges, what were the scores.
Investigation findings. What you learned about why variance occurred.
Decision and reasoning. What action you took and why that action was appropriate.
Outcome impact. Whether the variance affected final results and how.
Why documentation matters
Consistency. Similar situations should receive similar treatment. Documentation creates precedent you can reference.
Defensibility. If results are questioned, documentation shows that variance was identified and handled appropriately, not ignored or handled arbitrarily.
Learning. Reviewing variance patterns across cycles reveals whether your criteria, calibration, or judge selection need adjustment.
Accountability. Knowing that decisions are documented encourages thoughtful handling rather than expedient shortcuts.
Communication about variance
How you discuss variance, with judges, entrants, and external audiences, affects credibility.
With judges
Normalise disagreement. Help judges understand that some variance is expected and healthy. Judges who fear their scores are “wrong” may self-censor rather than provide honest evaluation.
Explain aggregation. When judges know their scores will be combined with others, they can focus on their honest assessment rather than trying to guess what others will score.
Share patterns without blame. If calibration or variance analysis reveals patterns, share them constructively: “We noticed some variation in how judges are interpreting ‘innovation’, here’s clarification” rather than “These judges are scoring wrong.”
With entrants
Don’t overpromise consistency. Claiming that your judging produces identical scores from all judges sets expectations you can’t meet and invites challenge when variance becomes visible.
Frame aggregation positively. Multiple judges exist precisely because combining perspectives produces better outcomes than any single view.
Address concerns factually. If an entrant questions why their scores varied, explain your process: “Our judges evaluate independently, and some variation is normal.
Scores are aggregated across multiple judges to balance different perspectives.”
With external audiences
Describe your process, not perfection. Credible programs explain how they manage variance, not claim it doesn’t exist.
Highlight safeguards. Calibration, multiple judges, conflict management, and variance monitoring all support credibility. Mentioning these demonstrates thoughtful process.
Avoid defensive overstatement. Claiming “perfect consistency” or “unanimous agreement” invites scrutiny and makes actual variance seem like scandal. Honest description of careful process is more credible.
Learning from variance patterns
Variance isn’t just a problem to solve, it’s information about your program.
What variance patterns reveal
Criteria clarity. Consistent high variance on specific criteria suggests those criteria need clearer definition.
Calibration effectiveness. If variance patterns match pre-calibration patterns, calibration isn’t working well enough.
Judge fit. Judges who consistently diverge from peers may not be well-suited to your evaluation, or may be bringing valuable dissenting perspective that deserves attention.
Entry design. If certain entry formats or question types produce more variance, entry design may need adjustment.
Category definition. Variance concentrated in specific categories may indicate those categories have unclear boundaries or criteria.
Applying learning
Review variance patterns after each program cycle:
- Which criteria produced most variance? Do they need refinement?
- Which judges diverged most? Is their perspective valuable or problematic?
- Which entries were hardest to evaluate? What made them difficult?
- Did calibration reduce variance compared to previous cycles?
Use findings to improve criteria, calibration, judge selection, and entry design for future cycles. Programs that learn from variance improve over time; programs that just manage variance cycle after cycle don’t evolve.
Checklist
Handling score variance
Monitoring
[ ]Variance tracking in place during judging
[ ]Thresholds defined for flagging concerning variance
[ ]Patterns monitored across judges, entries, and criteria
Investigation
[ ]High-variance entries reviewed directly
[ ]Judge reasoning examined when available
[ ]Judge-level patterns analysed
[ ]Potential conflicts checked
[ ]Judge consultation conducted if appropriate
Response selection
[ ]Response matched to variance cause
[ ]Aggregation used for normal perspective differences
[ ]Normalisation considered for systematic scale differences
[ ]Additional scoring used for decision-critical uncertainty
[ ]Score removal reserved for clearly anomalous cases
[ ]Deliberation used for high-stakes final decisions
Documentation
[ ]Variance cases documented with investigation findings
[ ]Decisions recorded with reasoning
[ ]Outcome impact noted
[ ]Records retained for consistency and learning
Communication
[ ]Judges understand that some variance is normal
[ ]Entrant inquiries answered factually
[ ]External communications describe process honestly
Learning
[ ]Post-cycle variance patterns reviewed
[ ]Criteria refinements identified
[ ]Calibration effectiveness assessed
[ ]Improvements planned for future cycles
Conclusion
Some judge disagreement is healthy, it reflects the reality that quality assessment involves judgement, and reasonable experts weigh factors differently. The challenge isn’t eliminating variance but distinguishing what’s normal from what’s concerning. Modest differences across multiple judges usually reflect legitimate perspective variation, and the right response is simply letting aggregation work. But extreme divergence, single-judge outliers, or patterns that correlate with potential conflicts warrant investigation before you act. Start by reviewing the entry itself, examine judge reasoning if available, look at patterns across each judge’s full set of scores, and check for undisclosed relationships. Once you understand what’s driving variance, match your response to the cause: accept and aggregate for normal perspective differences, apply statistical normalisation for systematic scale usage patterns, request additional scoring when critical decisions hang in the balance, or use deliberation for high-stakes final determinations. Remove outlier scores only when they’re clearly anomalous and unexplained, it’s a consequential action that should be documented and used sparingly.
Whatever approach you take, document it. Record which entry, which judges, what scores diverged, what you learned from investigation, what action you took and why, and whether it affected final results. This documentation creates consistency across similar situations, defensibility if results are questioned, and learning that improves future cycles. Variance patterns are information about your program, consistent high variance on specific criteria suggests those criteria need clearer definition, variance that matches pre-calibration patterns means calibration isn’t working well enough, and variance concentrated in specific categories may indicate unclear boundaries. Programs that treat variance as diagnostic information rather than just a problem to manage improve over time, while programs that just firefight cycle after cycle never evolve.
→ How to calibrate judges for consistent scoring
→ How to write judging criteria that judges can actually use
→ Managing conflicts of interest in awards judging
→ Judging & Scoring Best Practices
Frequently Asked Questions
udges disagree for several distinct reasons:
- Legitimate perspective differences (e.g., startup vs. enterprise lens)
- Different interpretations of criteria (e.g., what counts as “innovation”)
- Scale usage habits (lenient vs. strict scorers)
- Entry ambiguity (mixed strengths and weaknesses)
- Engagement or quality issues (superficial scoring)
- Undisclosed conflicts of interest
Some causes, such as perspective differences, are healthy and expected. Others, like conflicts or disengagement, require intervention to protect the credibility of results.
Investigation is warranted when variance could materially affect outcomes, such as finalist selection or winner determination, or when the pattern suggests something beyond normal judgement differences.
Triggers for investigation include:
- Scores at opposite ends of the scale
- A single judge repeatedly diverging from all others
- Sudden inconsistency in a judge’s scoring
- Possible undisclosed relationships with an entrant
- Fatigue patterns toward the end of judging
In these cases, review the entry, examine written justifications, analyse judge-level patterns, and check for conflicts before deciding on any corrective action.
When a judge consistently scores more generously or harshly across all entries, the issue is often systematic scale usage rather than flawed judgement. In these situations, statistical normalisation, such as z-score adjustment, can align scoring patterns without overriding individual perspectives.
Normalisation works best when:
- There is enough data to identify reliable patterns
- Judges evaluated comparable distributions of entry quality
- The method is documented and defensible
This approach preserves independence while correcting structural scoring differences.
Yes, but only in rare and clearly justified situations. Removing a score may be appropriate when it is dramatically different from all others, lacks credible reasoning, and appears tied to disengagement or conflict of interest.
- Because score removal directly affects outcomes, it should:
- Be based on documented investigation
- Follow consistent criteria
- Be used sparingly
- Be formally recorded with reasoning and impact noted
Removing scores simply because they are inconvenient undermines judge independence and program credibility.
For high-stakes decisions, independent scoring can be complemented with structured deliberation. After scores are submitted, judges can discuss high-variance entries to clarify reasoning and reconsider factors they may have underweighted.
Other options include:
- Assigning an additional judge
- Requesting re-evaluation with clarified guidance
- Moving an entry to a better-fitting category if misalignment is driving disagreement
The goal is not forced consensus but informed judgement that reflects both independent assessment and thoughtful discussion.
Documenting variance creates consistency, defensibility, and learning for future cycles. For each case, record:
- The entry and judges involved
- The scores and magnitude of divergence
- Investigation findings
- The action taken and reasoning
- Whether the variance affected final outcomes
This record supports transparent responses to challenges, ensures similar cases are handled consistently, and reveals patterns that may signal unclear criteria, ineffective calibration, or category design issues.
Variance is diagnostic information. Consistent high variance on specific criteria may indicate unclear definitions. Persistent divergence despite calibration suggests calibration needs strengthening. Variance concentrated in certain categories may signal boundary or fit issues.
Post-cycle analysis should review:
- Which criteria produced the most disagreement
- Which judges diverged most and why
- Which entries were hardest to evaluate
- Whether calibration reduced variance compared to previous cycles
Programs that treat variance as insight, rather than just a problem, define criteria, improve judge selection, and strengthen credibility over time.