Judging is where awards earn their reputation or quietly lose it, because however well a program is marketed and however impressive its winners look, trust comes down to a single question: was the decision fair?
That question gets answered almost entirely through your judging process, which is what makes judging the most fragile part of any program. And when outcomes are questioned, the judges are usually the first people blamed, though in reality most judging problems aren’t caused by poor judgement, bias or a lack of expertise at all. They’re caused by structural failures, by systems that place unreasonable demands on judges while giving them very little to work with.
This guide covers how to design judging that produces consistent, defensible results, from criteria and scoring models through to judge selection, briefing and the experience you give the people doing the work.
On this page
- Why most judging problems are system failures
- How judges actually evaluate entries
- Designing criteria that work
- Choosing a scoring model
- How many judges you need
- Selecting the right judges
- Briefing and preparing them
- Managing bias and inconsistency
- Why the judge experience matters
- Should judges give feedback to entrants?
- Common judging mistakes
- The judging checklist
Why most judging problems are system failures
Your judges are experienced professionals who bring industry knowledge, critical thinking and context to the work. What they’re rarely given is enough structure to apply that expertise consistently across dozens or hundreds of entries, and the pressures show up fast:
- Large volumes of submissions with almost no standardisation between them
- Vague or overlapping criteria that are open to interpretation
- Entry quality and formatting that varies wildly from one submission to the next
- Timelines tight enough to force rapid comparison
- No shared understanding of what “excellent” actually means in this category
Under those conditions even the most capable judge is forced to rely on shortcuts, so decisions become harder to explain, scoring diverges between panel members, and confidence in the result weakens, none of which is anybody’s fault individually.
The fragility sits in the gap between what programs expect and what they’ve designed, because organisers often assume that expert judges will compensate for unclear criteria, poorly structured entries and a weak scoring framework. In practice that puts unfair responsibility on individuals and introduces inconsistency into something that ought to be defensible.
How judges actually evaluate entries
Designing a fair system means understanding how judging really happens, because judges rarely assess entries in isolation, they work by comparison and pattern recognition, usually under time pressure and considerable cognitive load.
It starts with a scan. Most judges open an entry looking for clarity, relevance and some signal that the entrant understood the category, and a well-structured entry stands out immediately, not because it’s better, but because it’s easier to understand. A poorly structured one demands real effort before its merit can even be assessed.
Comparison becomes unavoidable. As judging progresses the question stops being “is this good?” and becomes “is this stronger than the others?” Without consistent information across entries those comparisons get uneven, because judges are left remembering details, flipping between formats and mentally normalising vastly different levels of detail.
Then fatigue sets in. Cognitive load builds with every unclear entry and every vague claim, and as it does, judges lean harder on their anchors, the defined criteria, the scoring scale, the benchmarks. Where those anchors are missing or poorly defined, judgement becomes progressively more subjective even among judges acting entirely in good faith.
And interpretation gaps widen. A term such as “innovation” might mean novelty to one judge and measurable impact to another, so scores diverge, not because the panel disagrees about quality, but because they’re scoring different things without realising it.
Designing criteria that work
Your criteria are the lens every entry gets viewed through, so when the lens is clear judges see the work consistently, and when it’s distorted even strong entries can be misjudged.
Start with restraint. A short list of clearly defined criteria outperforms a long list every time, and each one should exist for a reason and measure something distinct, because if two criteria overlap they’re double-counting the same behaviour and adding confusion rather than rigour.
Prioritise clarity over clever wording. Each criterion should describe exactly what judges are being asked to assess, with a short guidance note explaining what a high-scoring response looks like, which is what closes the interpretation gap before it opens.
Tie criteria to the category’s purpose. A category built to reward technical execution shouldn’t lean on subjective measures, while one celebrating creativity or impact needs room for qualitative judgement. Where criteria and category intent pull in different directions, the outcomes feel arbitrary to everybody.
Weight deliberately. Not every criterion deserves equal influence, and assigning intentional weightings helps judges prioritise what matters while preventing a secondary factor from swinging a result. Communicate those weightings and apply them consistently.
Design for scale. A judge may work through dozens of entries in one category, so criteria that demand extensive interpretation or long written justification increase fatigue and erode consistency as the pile goes down.
| Vague criterion | Clear criterion |
|---|---|
| “Innovation” | “Introduces a new approach, method or solution that did not previously exist in this market” |
| “Impact” | “Demonstrates measurable outcomes with supporting evidence, such as metrics, testimonials or before-and-after comparison” |
| “Excellence” | “Executes against stated objectives with high quality and minimal errors” |
| “Leadership” | “Shows evidence of influence on a team, organisation or industry beyond individual contribution” |
Good criteria don’t restrict expert judgement. They focus it.
Choosing a scoring model
Your scoring model determines how individual judgements become outcomes, and although the mechanics sit behind the scenes, their effect is visible in every result. The governing principle is alignment: the model should support the category’s purpose and its criteria rather than distorting them.
| Model | How it works | Best for | Watch out for |
|---|---|---|---|
| Numeric | Judges score within a set range, such as 1–5 or 1–10, for each criterion, aggregated by average or total | Categories with several criteria, larger panels, and any need for quantitative comparison | Arbitrary scoring where scale guidance is missing, some judges score generously and others conservatively, which makes averages misleading |
| Weighted | Numeric scoring with greater influence assigned to the more important criteria | Categories where some criteria genuinely matter more, helping core objectives drive the outcome | Only works when judges understand what the weights represent and apply them consistently |
| Ranked | Judges order entries relative to each other rather than scoring each in absolute terms | Smaller categories where differentiation matters more than measurement, and it removes scale bias entirely | Gets unwieldy as entry numbers grow, and close rankings are hard to explain afterwards |
| Pass/fail | Entries must clear a minimum standard to proceed, usually as a first-round filter | Non-negotiable requirements such as eligibility, completeness and compliance | Feels harsh where the threshold wasn’t communicated clearly, use it for screening, not for merit |
Whichever model you choose, be transparent about aggregation, because judges and organisers both need to understand how individual scores combine and what happens when two judges score the same entry very differently.
How many judges you need
There’s no universal answer, though the question matters more than most organisers realise, and four things shape it: your entry volume, the complexity of the category, how much consistency you need, and what you can realistically manage in terms of recruitment and briefing.
| Program size | Suggested approach |
|---|---|
| Small, under 50 entries | 3–5 judges per category |
| Medium, 50 to 200 entries | 5–7 judges per category, possibly split across rounds |
| Large, 200 or more | Two-round judging with a screening panel feeding a final panel |
More judges per entry improves reliability but adds coordination work, so there’s a real trade-off rather than a target to hit. Use odd numbers where you can, since that avoids ties needing a tiebreaker, and don’t assume every judge needs to see every category, matching judges to categories by expertise improves the quality of assessment and spares everybody the entries they’re not equipped to evaluate.
And if you also need to pick a scoring model for each category:
| Situation | Model |
|---|---|
| Multiple criteria, high volume | Numeric with weighting |
| Small shortlist, final round | Ranked |
| Eligibility or compliance check | Pass/fail threshold |
| Need to emphasise particular priorities | Weighted scoring |
| Simple category, few criteria | Numeric, unweighted |
Selecting the right judges
The credibility of your judging rests on who’s doing it, so selection deserves more thought than it usually gets.
Expertise has to be genuine and specific. Seniority isn’t the same as qualification, and a marketing executive may be entirely the wrong person to judge a technical development award however impressive their title. The test worth applying: would entrants respect this person’s judgement in this particular category?
Diverse panels make better decisions. Consider professional background and experience, organisation size and type, geography where it’s relevant, and the usual dimensions of gender and age, because a homogeneous panel risks groupthink and shared blind spots that nobody in the room can see.
Conflicts need declaring before you start. Judges shouldn’t evaluate entries from their own organisation, from a direct competitor where bias is likely, or from personal connections such as friends, former colleagues or clients.
| Internal judges | External judges |
|---|---|
| Know your awards and criteria well | Bring independent credibility |
| Easier to coordinate | Need more briefing |
| Risk of perceived bias | Perceived as more objective |
Many programs resolve this by using internal judges for screening rounds and external ones for the final decisions, which gets the coordination benefit where it matters and the independence where it’s visible.
Briefing and preparing them
Even experienced professionals need preparation, so never assume a judge will work it out from the interface.
- Criteria with definitions, not just the labels, but what each one means and what a good response looks like
- Scoring guidance, how to use the scale and what each level represents
- Timeline and expectations, how many entries, roughly how long each takes, and when their scores are due
- Entry format orientation, how a submission is structured and where the key information sits
- The conflict policy, what to do if they recognise an entrant halfway through
For larger programs, run a calibration session where the panel scores two or three sample entries together and then compares notes, because that surfaces interpretation differences while there’s still time to resolve them.
Managing bias and inconsistency
No judging process is free of bias, and the goal isn’t to remove human judgement, it’s to design a system where unnecessary influences carry less weight.
Familiarity bias
A well-known brand or individual benefits from recognition regardless of what the entry actually says.
Presentation bias
A polished, professionally written entry outscores a substantively stronger but less slick one.
Anchoring bias
An unusually strong or weak entry read early shifts how everything after it gets scored.
Affinity bias
Judges unconsciously favour work that resembles their own experience, style or preferences.
None of these is intentional, and that’s precisely the problem, without structure they shape outcomes anyway. Six safeguards do most of the work:
- Clear criteria with shared definitions
- Defined scoring ranges with descriptors at each level
- Category-specific judge assignment based on expertise
- Randomised entry order, where your platform allows it
- Multiple judges per entry, to balance out individual variation
- Rules applied consistently to every entry without exception
When the framework is clear and explainable, you can respond to scrutiny confidently, your judges feel supported rather than exposed, and entrants can see how the decision was reached. Bias isn’t solved by taking humans out of the process, it’s managed by designing a process that supports fairness and accountability.
Why the judge experience matters
Judges are central to your credibility, and yet their experience is usually an afterthought. Most of them are senior people contributing time alongside demanding jobs, so when the process is confusing, repetitive or administratively heavy, frustration arrives quickly, and it affects both the quality of their decisions and whether they say yes when you ask again next year.
A fatigued judge makes faster and less nuanced decisions, communication gets strained, and feedback quality drops. When judges disengage entirely you lose one of your most valuable assets, which is trusted expertise you didn’t have to pay for.
The reverse is just as true, since judges who feel supported are more consistent, more thoughtful and more invested. Good judge experience means:
- Clear expectations set from the outset
- Criteria and scoring that are easy to apply in practice
- An interface that minimises clicks and repetition
- A timeline that respects the availability they actually have
- Transparency about how their scores get aggregated
- Genuine recognition of the contribution they’ve made
Judges who feel respected speak well of your program, come back, and recommend it to their peers. Programs invest heavily in attracting entrants and promoting winners while overlooking the people making the decisions, which is a missed opportunity — your judges aren’t only evaluators, they’re partners in your credibility.
Should judges give feedback to entrants?
Feedback is among the most requested and most badly handled parts of awards programs, and it’s a genuine decision rather than an obvious yes.
The case for
It adds value for entrants beyond winning or losing, demonstrates respect for the effort they put in, encourages improvement and repeat participation, and differentiates you from programs that offer nothing at all.
The case against
It significantly increases judge workload, risks inconsistent or unhelpful comments, creates a route to disputes and complaints, and can reveal more about a judge’s preferences than about the entry’s quality.
If you do offer it, four things keep it manageable. Structure it, because open comment boxes produce wildly variable quality, prompt for strengths, areas for improvement, and one suggestion. Set expectations by telling entrants it’s summary-level and advisory rather than comprehensive or a basis for appeal. Make it optional for judges, or limit it to shortlisted entries, since mandatory feedback on every submission is where fatigue comes from. And review it before it goes out, because one careless comment can do disproportionate damage.
If you don’t offer it, say so clearly before entries open so nobody is surprised, and consider publishing aggregate insights instead, the common strengths and weaknesses across all entries adds real value without any individual burden.
Common judging mistakes
Three problems come up repeatedly that aren’t covered by getting your criteria and scoring right.
Mixing qualitative and quantitative without deciding why. Open comments add valuable nuance, but combined with numeric scores and no defined role they create confusion about which one actually decides anything. Define the purpose of each element: is the comment for the organiser, for the entrant, or for the record?
Changing the rules during judging. Adjusting criteria, weightings or thresholds mid-process raises immediate questions about fairness, even where the reason was entirely practical. Once judging starts, consistency isn’t optional, lock the rules.
Treating judges as interchangeable. Differences in expertise, background and category familiarity all affect how somebody evaluates work, and ignoring that flattens the insight you recruited them for. Match judges to categories deliberately, and weight expertise where it’s warranted.
The judging checklist
Designing fair, defensible judging
Criteria design
- Criteria are few, focused and non-overlapping
- Each criterion has a clear definition and guidance
- Criteria align with the category’s purpose
- Weightings are intentional and communicated
Scoring model
- Model matches the needs of the category
- Scale descriptors provided for every level
- Aggregation method documented
- Judges understand how scores become outcomes
Judge selection and preparation
- Judges have relevant expertise for the categories assigned
- Panel is appropriately diverse
- Conflict of interest policy in place
- Judges briefed on criteria, process and timeline
- Calibration session held, for larger programs
Process integrity
- Rules locked before judging begins
- Same standards applied to every entry
- Conflicts declared and managed
- Judge workload is realistic
Judge experience
- Interface is clear and efficient
- Timeline respects judges’ availability
- Judges are thanked and recognised
Feedback, if offered
- Feedback is structured rather than open-ended
- Expectations set with entrants before they submit
- Comments reviewed before delivery
Design the system, then trust the judges
Almost everything that goes wrong in judging traces back to a decision made before any judge saw an entry, criteria left undefined, a scale published without descriptors, a panel recruited without regard to which categories they’d actually assess.
Fix those and the judging itself largely takes care of itself, because you’ll have given experienced people a clear question, a shared standard and enough time to answer properly. That’s all most judges need, and it’s more than most programs give them.
Continue learning
- Awards strategy and design — how to structure categories, eligibility and judging models from the ground up.
- Entries, storytelling and quality control — how to design entry processes that give judges clear, comparable information.
- Public voting and awards engagement — when and how to bring in audience participation without compromising credibility.