Say you’ve written two versions of a project update. Both sound professional, but you need to pick one before you send it. You prefer the smoother version, then notice that it never says what happens next.
The other version names the next step, but claims something is finished that you haven’t checked. Choosing well means spotting those differences and deciding what they mean for the reader. You may need to fix an option before you use it.
When it’s easy to produce several options, choosing between them can become the harder part. That matters when the options differ in important ways that are hard to spot. More practice is useful only when it works on something your work actually needs.
This module is for working adults judging work in a field they know, with clear standards for quality and a way to get feedback.
What you’ll get from this module
In this module
By the end of this module, you’ll have a clearer way to explain a choice and find out whether your judgment holds up. We’ll link research on comparing essays, practice at rating, training for examiners, people and AI working together, and how music gets found. You’ll think about what the findings mean, pick a situation from your daily life and check one decision against feedback. You’ll finish with a better reason, a next thing to learn or a clear gap to close.
Ask Jing
Become a Jingius.
What makes a judgment worth trusting?
Classroom experiment: comparison changes what you notice.
Renske Bouwer and colleagues (2018) studied how essays were judged in a Belgian business-management class. The analysis included 40 students. Each was randomly assigned either to compare pairs of English essays or to score single essays against criteria. Both groups got the same criteria, written by the teacher.
Students in each group judged the essays consistently, and the two approaches mostly agreed about which essays were better. Comparing pairs shifted feedback about what needed improving toward content and structure, and away from grammar and vocabulary. The study couldn’t show whether comparing essays led to better writing later than scoring them did.
Why it matters: The useful difference is where attention went while judging. A comparison can bring an important difference between two pieces into view, even when both approaches give usable judgments. Here, comparison changed what people criticised. It didn’t show better writing later.
Read the classroom study of essay comparison.
Rating-training study: practice and feedback contribute differently.
Noticing a difference leaves another question: was your judgment accurate? Neil M. A. Hauenstein and Maureen E. McCusker (2017) looked at which parts of training helped people rate performance more accurately. They used a carefully designed comparison group.
Their published abstract reports that repeated practice at rating improved accuracy, and feedback made that improvement bigger. Teaching content made raters agree more with each other, without making them more accurate than the control group.
Agreement tells you that people are judging in a similar way. Accuracy is about whether their judgments match a suitable standard. The findings make that split useful: people can rate more alike without rating more accurately.
Why it matters: The abstract supports those findings about each part without saying how big the improvement was. The practical lesson is to link your judgments to information that can show your mistakes.
Read the study abstract on rating practice and feedback.
Randomized field trial: tougher grading wasn’t more accurate grading.
A real assessment shows why the result needs checking. Gunther Weitz and colleagues (2014) studied 21 doctors who acted as examiners at University Hospital Lübeck. Eleven were randomly assigned to a 90-minute session with videos, ratings, discussion and feedback.
The doctors then graded students doing physical examinations on people acting as patients, who followed set instructions. Three observers rated recordings on their own, without knowing which examiners had been trained. The middle of those three ratings was used as the standard for accuracy.
Trained examiners graded more strictly, but the study found no clear improvement in the accuracy of their ratings. During training, examiners rated separate parts of performance. In the real assessment, they gave one overall grade. This was a small trial in one setting, and the observers’ ratings were not a perfect standard.
Why it matters: So a change in how someone judges is not enough to show improvement. Being tougher can feel like progress while the question of accuracy stays open. The useful test is about the judgments you need to make in the real task.
Read the field trial of examiner training.
Systematic review and meta-analysis: the comparison you choose changes the answer.
The same question applies when a person and AI work together. Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone (2024) combined results from 106 experiments published between January 2020 and June 2023.
Across the tasks studied, people and AI together did better than people alone, on average. But they did worse than the better of the two working alone: the person or the AI. Results varied a great deal across tasks and settings.
These comparisons answer different questions. Help can improve a person’s work while the joint process still falls short of the stronger of the two. Adding a reviewer doesn’t, on its own, show that a decision got better.
Why it matters: Tasks where people made something looked better than tasks where they decided something. Even for making tasks, though, the review couldn’t show that the pair beat the better one working alone, on average. The findings support checking the actual combination, rather than assuming that a person’s input or automation gives the best judgment.
Read the review of human–AI combinations.
Working-paper analysis: reaching people can be a separate problem.
Even well-chosen work needs to reach its audience. Luis Aguiar and Joel Waldfogel’s 2018 working paper looked at Spotify streams and playlist placement, including daily song data from 26 countries during 2016–2017.
The researchers used several comparisons. These included changes when songs were added to or removed from playlists, chart cutoffs and differences across countries. They estimated that being on a playlist increased streaming and helped new music get found. The methods try to separate the effect of playlists from how popular songs already were. The estimates depend on those comparisons’ assumptions, and on the songs and playlists the data covered.
Why it matters: This is a lesson about getting work to people: what a platform picks can affect which work people come across. It doesn’t show that the songs picked were truly the best. If good work isn’t reaching the people it is meant for, getting better at choosing between drafts may leave the main problem untouched.
Read the working paper on playlists and discovery.
What this means for you
Treat judgment as something you can explain and check in a particular task. Start with work you understand, keep the purpose in view and look for signs that your choices are getting more useful.
Name the difference that matters
Compare examples of different quality against a clear purpose. Ask which feature makes one option more suitable, and what that feature leads to. A preference is easier to check when you can point to the work itself.
In the made-up project update, smooth wording and a clear next step are different qualities. If the reader needs to know what happens next, the missing step matters. If a claim that work is finished isn’t backed up, a clearer layout doesn’t make that claim usable.
You could keep the useful layout and fix the unsupported claim. Explain the choice by the need it meets and the fault it still has. This stops “I like this one” from passing as a full judgment.
You still need to know the subject, and how to produce suitable work, so you can judge it against standards that mean something. If you don’t know what a reliable update needs to contain, learning that is part of the work ahead.
Make a choice before checking the feedback
Write down your choice, and the feature that decided it, before you look at anyone else’s judgment. Then compare your reasoning with feedback from someone who knows the task and what it needs, or with a result you can see.
Look for a clear match or mismatch. Did you spot the important fault? Did you reject something that actually met the need? Was your reason about the purpose, or about a feature that mattered less?
In the update example, feedback could confirm that the missing next step left the reader not knowing what to do. It could also show that you misread the claim that the work was finished. Change your explanation to match what you learned.
If you don’t have a trustworthy reference, write down what is missing. A confident choice, or agreement with another person, doesn’t settle whether the choice fits the task.
Check performance on unfamiliar examples
Try examples you haven’t already talked through. Use the same standards and check your choices, and the faults you missed, against suitable feedback. Notice how long the judgments take as well as how good they are.
The examiner study shows why becoming stricter is not a sign you’ve finished. The feedback study separates agreement from accuracy. For your practice, the useful question is whether you can make and explain better-supported choices on work you haven’t learned by heart.
If the improvement doesn’t carry over, look at the standards, the feedback and how well the practice matches the real work. Change the practice before you commit more to it.
Evaluate assistance by what it adds
If you use AI or another reviewer, check the choices you make together and the mistakes you miss. Use new examples that reflect your real work. Use a suitable reference, and think about how costly different mistakes would be.
Compare the options that matter: your choice alone, the helper’s choice alone where that makes sense, and the choice you make together. Doing better than you would alone doesn’t mean the pair is the strongest option.
If you can’t tell which choice is better, keep that uncertainty in view. Liking the explanation, or reviewers agreeing, can’t replace checking what the decision leads to.
Put learning effort where your work gets stuck
Use your real difficulty to choose what to practise next. You don’t need to treat every problem as poor judgment.
| What you can demonstrate | Useful next learning target |
|---|---|
| You can’t yet produce work that meets the requirements. | Build what you know about the subject and your skill at making the work. |
| You can produce plausible options but can’t reliably select, diagnose or repair them. | Practise comparisons that you check against feedback. |
| Your selected work meets requirements but doesn’t reach its intended users. | Look into how the work reaches people. |
For the project update, not being able to explain the facts points to subject knowledge. Trouble spotting missing information points to judging the work. A good update that never reaches its reader points to how it is delivered.
Before you put in much more time, use new tasks to check that you’ve picked the right thing to learn. If you can’t yet tell where the difficulty is, name the evidence you need. An open question is a useful result when it stops you committing to something unsupported.
How about your daily life?
Where do you choose between pieces of work that all look reasonable? You might compare project updates, drafts of a presentation or explanations for people you work with.
Where do you struggle to explain a preference, miss an important fault or keep improving work that doesn’t reach anyone? Choose something you know well enough to name useful standards and get feedback.
Write down possible situations. One good idea is enough.
| A situation in my daily life | What I want to judge or understand better |
|---|---|
| ____________________ | ____________________ |
| ____________________ | ____________________ |
My chosen situation: ____________________
Carry this situation into the exercise. You can change it if the standards or feedback you need aren’t available.
Check one choice and your reason
Start with your chosen situation. If you came straight here, choose work you know, with clear standards and a way to get feedback, such as comparing project updates. If these are missing, name what you need before you treat a choice as checked.
- State the purpose and quality requirement. Write what the work must help the person using it do. Name a feature you can actually look at.
Purpose and requirement: ____________________ - Compare examples and make your choice. Use examples that differ in quality. Before you see feedback, write down the option you prefer, the feature that decided it and what that feature leads to. Name any fault that still needs fixing.
Choice, reason and unresolved defect: ____________________ - Check against credible feedback. Compare with the judgment of someone who knows the work, or with a result you can see. Write down what matched and what you missed. If the reference can’t settle the choice, write what information is missing instead of marking it correct.
Feedback, mismatch or missing information: ____________________ - Revise and try unfamiliar work. Update your reason or fix the option as needed. Judge a new example using the same standards, then check it. Compare how well your choices meet the need, which faults you miss and how long you take.
Revised reason and unfamiliar-example result: ____________________ - Choose the next learning target. Work on knowing the subject and making the work if you can’t deliver good enough work. Work on judging if choosing or fixing is the problem. Look into how work reaches people if good work isn’t reaching the people it is for.
If the result is unclear, name the next useful check. Before you do more of this practice, look for improvement on new tasks. If you use help, compare what it adds with the other options that matter, and with what mistakes would cost.
Next learning target or unresolved question: ____________________
Ask Jing
Become a Jingius.
Your next step
Finish with a reason you can explain, what the feedback showed and a next step tied to the real difficulty. If a standard or reference you need is missing, your result is that clear gap.
A note on the evidence
The training evidence is about particular tasks and measures. Applying it to everyday work means checking whether the practice helps in that setting. Improving at these rating tasks doesn’t show lasting expertise across unrelated kinds of work.
The playlist findings here come from the 2018 working paper. The human–AI review covers experiments published up to June 2023, and the results varied a lot. None of this shows that human judgment will always be scarce or earn higher pay.
How we prepared this module
This module links research on judging, training, people and AI working together, and getting work to people with a practical comparison exercise. A made-up project-update example explains how a requirement can guide a choice and show a fix that is needed.
Sources
- Renske Bouwer, Marije Lesterhuis, Pieterjan Bonne and Sven De Maeyer. (11 October 2018). Applying Criteria to Examples or Learning by Comparison: Effects on Students’ Evaluative Judgment and Performance in Writing. Frontiers in Education, 3, 86. Published article; Methods, “Participants” and “Materials and procedure”; Results, “Quality of peer assessment” and “Quality of writing performance”; Discussion.
- Neil M. A. Hauenstein and Maureen E. McCusker. (September 2017; online 4 August 2017). Rater training: Understanding effects of training content, practice ratings, and feedback. International Journal of Selection and Assessment, 25(3), 253–266. Publisher abstract, results sentences.
- Gunther Weitz, Christian Vinzentius, Christoph Twesten, Hendrik Lehnert, Hendrik Bonnemeier and Inke R. König. (17 November 2014). Effects of a rater training on rating accuracy in a physical examination skills assessment. GMS Zeitschrift für Medizinische Ausbildung, 31(4), Doc41. Published English article; Methods, “Intervention” and “Video-based re-assessment”; Results, “Effect of training on grading and accuracy, effect of experience on accuracy”.
- Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone. (28 October 2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8, 2293–2303. Published article; Results, “Overall levels of human–AI synergy”, “Heterogeneity of human–AI synergy” and “Moderators of human–AI synergy”.
- Luis Aguiar and Joel Waldfogel. (June 2018). Platforms, Promotion, and Product Discovery: Evidence from Spotify Playlists. National Bureau of Economic Research (NBER), Working Paper 24713. Working-paper version; Introduction, pp.1–4; “Data”, pp.8–9; playlist identification analyses, sections 4–5.
