Say you’re getting a submission ready. You ask an answering system whether you’ve included everything. It gives you a clear list. You check your documents against that list and feel ready to send them.
But what if the list leaves out an attachment you have to send? Every item you checked could be correct, and something you need could still be missing. Checking what an answer says is one job. Checking what the task requires is another.
This module is for adults who use an answering system and don’t know whether its confidence is a reliable sign of accuracy for their own task.
What you’ll get from this module
In this module
By the end of this module, you’ll know which answers need checking and what evidence could settle them. We’ll link experiments to day-to-day decisions. They cover automated advice, uncertainty wording, explanations and confidence displays. You’ll choose a situation from your daily life. You’ll finish with a checked answer, a correction or a doubt you have stated clearly.
Ask Jing
Become a Jingius.
What changes when an answer sounds convincing?
Research report: help and error can exist side by side.
Linda J. Skitka’s 1998 report separates two kinds of error. Commission means following wrong automated advice even when other information is at hand or points the other way. Omission means missing a problem when the aid doesn’t flag it.
Its first study had 80 undergraduates. They flew eight simulated flights with tracking and monitoring tasks. Some had an automated aid. Others worked without it. These were students in a simulation, not trained pilots.
When the aid missed events, the people with the aid missed more of those events than the people without it. People with the aid also followed wrong directions. Yet across all 100 events, the aided group gave more correct responses overall.
That mix matters. A useful aid can raise overall performance and still leave certain failures that deserve attention. If you treat the tool as all helpful or all untrustworthy, you miss that difference.
Why it matters: The two errors also point to different questions. Is the advice right? And has something important been left unsaid? An answer can look complete because nothing in it tells you what is missing.
Read the research report on automated decisions.
Randomized experiment: changing the uncertainty while keeping the information.
Sunnie S. Y. Kim and colleagues’ 2024 study asked whether uncertainty wording changes how people use an answer. The researchers studied responses from 404 US online crowdworkers. Each was put in one of four groups at random.
They answered eight medical yes/no questions. The system’s answers were right on four questions and wrong on four. The versions that expressed uncertainty otherwise held the same information as the version without it.
First-person wording presents the system itself as unsure. General wording describes uncertainty without using personal pronouns.
Compared with no uncertainty wording, first-person uncertainty made people agree with the system less. Their answers were also more accurate. General uncertainty wording made no statistically significant difference to those two results.
The study found no significant differences in how much people used sources across the three groups that got system answers. So a change in agreement isn’t evidence that people carried out an outside check.
Why it matters: An answer can change how willing you are to accept it without giving you better evidence that it is right. Here, the wording changed how much people relied on the answer. The information under it stayed the same. The results are about this limited answering task, with its fixed mix of right and wrong answers.
Read the experiment on uncertainty wording.
Experimental study: explanations that make checking easier.
What would make a check more useful? Helena Vasconcelos and colleagues’ 2023 research looked at explanations in a maze task with simulated advice.
Study 1 had 340 crowdworkers. One version gave a prediction alone. The other also highlighted a route on the maze. The highlight gave people something they could check against the maze itself.
In hard mazes, route highlights made people accept wrong advice less often. No drop was detected in easy or medium mazes.
Why it matters: The value of the explanation here was concrete. It put a proposed route in front of people to check. A person could see whether the route worked, instead of having only a proposed answer. That is a different kind of help from getting more words in support of a conclusion.
Read the study of checkable maze explanations.
Controlled online experiment: an explanation can leave the answer open.
A different task shows why an explanation alone isn’t enough. Zelun Tony Zhang and colleagues’ 2024 study looked at 200 people. Their task was to tell people’s professions from biographies.
They were adults with degrees, living in the US, UK or Canada. English was their first language. They got advice with or without explanations of the features behind it. They were not told straight away whether their answers were correct.
As decisions got harder, agreement with wrong advice rose faster when explanations were shown. For the hardest decisions, agreement was higher with explanations. The researchers also looked at trials with wrong advice. They asked whether explanations changed how accuracy varied with difficulty. That result was not statistically significant.
A biography can fit several professions. A maze route is different, because you can inspect whether it is correct. An explanation can offer reasons and still leave you unable to tell whether those reasons settle the answer.
Why it matters: Together, these studies make the question more specific. What does the explanation let you check? The answer depends on the task and on the information it gives you.
Read the experiment on explanations for profession decisions.
Controlled experiment: using confidence is different from reaching the right answer.
Could a confidence score settle the problem? Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy tested this in 2020 with an income-classification task.
Their first experiment had 72 online crowdworkers, spread at random across eight conditions. They did training and test trials and were told whether they were correct. The researchers checked that the model’s confidence scores were a good guide to how often its answers were correct.
Showing the confidence helped people rely on the model more or less, in line with its confidence. But it did not significantly improve the accuracy of the decisions they made with its help.
Changing when you follow advice is only part of reaching a correct answer. You still need a way to reach a correct answer when you don’t follow it. If your own answer is unsettled too, changing whose answer you pick leaves that gap in place.
Why it matters: The score in this experiment had a checked link with accuracy. A confidence score that nobody has examined, in another task, doesn’t get that link just because it is given as a number.
Read the experiment on confidence and decision accuracy.
What this means for you
Start with what you plan to do with the answer. You can keep the convenience of the help and check the answers where a mistake would matter.
Start with what you intend to do
An answer you use to explore ideas does a different job from an answer you use to commit to something. Ask what would happen if this one answer were wrong.
Would it change what you do in a way that matters? Could it mislead someone you pass it to? Would the mistake be hard to correct later? These consequences give the check a purpose.
Casual exploring that is cheap to undo can go on without a full checking routine. Passing along a possible idea is also different from assuring someone that a factual claim is correct. Label a possible idea as an idea. Don’t pass it along as a checked fact.
Find the fact that would change your decision
When the use matters, find the fact, instruction or calculation your decision depends on. We can call this the hinge: if it changes, your decision changes.
Say the submission example turns on whether an attachment is required. The check that decides it is the original instruction about that attachment. Another confident answer about how complete the submission looks doesn’t settle the requirement.
Check the hinge against an original document that applies, a result you can observe or a calculation you can repeat. The system can help you find material or put your inputs in order. Being reassured by it again isn’t the check.
The maze and biography studies explain why this difference matters. A useful explanation gives you something you can compare with evidence. If you can’t tell whether its reasons hold, extra detail may leave the main question open.
Look for requirements the answer leaves out
Checking an answer sentence by sentence can deal with wrong statements. It can’t, on its own, reveal a key requirement that never appears in the answer.
For the made-up submission, compare the proposed contents with the original requirements. If the original instructions require an attachment that the answer leaves out, add it before you treat the submission as complete.
This deals with omission as well as commission. The first asks whether a needed condition is missing. The second asks whether you are about to follow advice that conflicts with the evidence you have.
Let the check decide what happens next
If the evidence supports the fact your decision depends on, go ahead, within what that evidence shows. If it contradicts the answer, correct the answer before you use it.
Sometimes the document is unclear, the result can’t be observed yet or the question needs a specialist’s judgment. In that case, keep the doubt in view. Flag the doubt in what you pass along, or put off the commitment that depends on it.
For the made-up submission, an unclear instruction leaves the attachment requirement open. You haven’t shown that the attachment isn’t needed. State that gap. Don’t quietly treat the answer’s silence as permission to leave the attachment out.
Confidence can help only when its link to accuracy has been shown for the task in question. It doesn’t replace the evidence you need to settle this one question. And rejecting the system’s answer doesn’t make your own answer correct.
How about your daily life?
Where do you use an answer to decide what to do or what to tell someone? You might summarize instructions, check a requirement or pass along a factual explanation.
Think about a situation where being wrong would matter. You can also pick out a casual use where a mistake would be easy to correct and would matter little. One useful idea is enough.
| A situation in my daily life | What I would like to understand or check |
|---|---|
| ____________________ | ____________________ |
| ____________________ | ____________________ |
Which situation would be most useful to explore now?
My chosen situation: ____________________
Carry that situation into the exercise below. Choosing it doesn’t yet mean every part of the answer needs checking.
Check one answer that matters
Start with your chosen situation. If you came straight here, choose an answer you’re thinking of using for a day-to-day action or passing along as fact.
Use this exercise when you don’t know whether the system’s confidence is a reliable guide to accuracy for that task. It helps you examine a decision. It doesn’t replace professional judgment on a specialist question.
- Name the use and its consequence. Write what you plan to do with the answer. Would an error change the action in a way that matters, mislead someone or be hard to correct later?
Use and consequence: ____________________
If this is casual exploring that is cheap to undo, you can stop without a full checking routine. If not, carry on with the part that matters. - Name the hinge. Find the fact, instruction or calculation that would change your decision if it were wrong.
My decision depends on: ____________________ - Check evidence and missing requirements. Look at the original document that applies, a result you can observe or a calculation you can repeat. When the decision depends on requirements, compare them with the answer. That way you can notice a condition it left out.
The system can help you find or organize information. Asking it to reassure you again doesn’t complete this step.
Evidence checked and any missing condition: ____________________ - Record the result before committing. If the evidence supports the hinge, go ahead within the limits of that evidence. If the evidence contradicts it, correct the answer. If the hinge stays unresolved, keep that doubt. Flag it in what you pass along, or put off the commitment that depends on it.
A specialist question that you can’t resolve stays unresolved here. Don’t turn a check you couldn’t make into a positive result.
Supported, contradicted or unresolved—and what I will do: ____________________
Ask Jing
Become a Jingius.
Your next step
Finish with a next action you can give a reason for, or a gap you have clearly named. The useful result is knowing what supports your decision and what still doesn’t.
A note on the evidence
These experiments used certain tasks and narrow groups of people. They measured different things: agreement with advice, use of outside sources, responses to confidence scores and correct decisions. A change in one doesn’t show a change in another.
The explanation studies also used different kinds of information. A maze route you can check and a biography that fits several answers don’t support one rule for all explanations. Use the difference to ask what a proposed check can actually settle.
How we prepared this module
This module links a research report and experimental studies. They cover automated advice, uncertainty wording, explanations and confidence displays. A made-up submission example makes one difference concrete: checking statements is not the same as checking requirements. The written exercise applies those differences to a situation you choose.
Sources
- Linda J. Skitka. (1998). Automation: Decision Aid or Decision Maker?. National Aeronautics and Space Administration (NASA), contractor report NASA/CR-1998-207767; NCC 2-986 final report. NTRS report; Introduction, printed p. 1; Study 1 Method, pp. 2–5; Results, p. 6.
- Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. (June 3–6, 2024). “I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust. ACM FAccT 2024 proceedings, Rio de Janeiro. Conference publication; §§3.1, 3.4–3.5, 4.1, 4.3 and 4.5; Table 1, PDF p. 7.
- Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael S. Bernstein, and Ranjay Krishna. (April 2023). Explanations Can Reduce Overreliance on AI Systems During Decision-Making. Proceedings of the ACM on Human-Computer Interaction, 7, CSCW1, Article 129. Author manuscript, arXiv 2212.06823; Study 1 §§5.4–5.6, printed pp. 129:14–16.
- Zelun Tony Zhang, Felicitas Buchner, Yuanting Liu, and Andreas Butz. (September 1–4, 2024). You Can Only Verify When You Know the Answer: Feature-Based Explanations Reduce Overreliance on AI for Easy Decisions, but Not for Hard Ones. Mensch und Computer 2024, 156–170. ACM conference manuscript; §§3.1, 3.4, 4.1–4.2, PDF pp. 2, 4–6.
- Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. (January 27–30, 2020). Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. ACM FAT* 2020 proceedings, 295–305. Author-hosted published PDF; Experiment 1 §§3.1.1, 3.1.3–3.1.4 and 3.2.1–3.2.2, PDF pp. 3–6.
