Say you’re choosing a transcription tool. It’s a trial you can end at any time. You’ve read several reviews, and the same good points keep coming up. You feel surer about your favorite than you did an hour ago. But you still don’t know whether it exports the file format you need.
Another good review and an answer about export formats do different jobs. One repeats reasons you’ve already heard. The other could rule the tool out, or clear a requirement you can’t do without.
That difference gives you a better way to think about research. Instead of asking whether you’ve read enough, ask what you still need to know for this one decision.
This module is for working adults making a specific decision they can reverse. Nothing in it should be unresolved if it could affect safety.
What you’ll get from this module
In this module
By the end of this module, you’ll be able to tell useful checking from reading that adds little to your decision. We’ll look at experiments on confidence, tax letters, insurance comparisons, investment documents and information search, along with a review of health decision aids. You’ll connect what they found to everyday choices, pick a situation from your own life and work through a written checkpoint. You’ll finish with a reason to decide, one specific question to look into, or a need for qualified help.
Ask Jing
Become a Jingius.
What does more information actually change?
Laboratory experiment: feeling surer isn’t the same as being right.
In 2008, Claire I. Tsai, Joshua Klayman and Reid Hastie asked how extra information changes what people predict and how sure they feel. In their first experiment, 30 University of Chicago students who knew football well picked the winners of college games. They got six statistics at a time, until they had thirty pieces of information.
After each batch, the students picked the winner again and rated how sure they felt. They grew significantly surer. How often they were right showed no significant upward trend. In the final round, confidence reached 79%, while 66% of predictions were correct.
Why it matters: The more statistics the students had, the surer they felt. Their picks didn’t clearly get better. This was a small sports experiment. Its useful lesson is simple: feeling more certain doesn’t, by itself, show that you have better grounds for a choice.
Read the experiment on information, accuracy and confidence.
Randomized field experiment: more detail can help or get in the way.
The next question is what information helps people do. In 2015, Saurabh Bhargava and Dayanand Manoli tested tax letters sent to 35,050 people in California. These people seemed to qualify for the 2009 Earned Income Tax Credit but hadn’t claimed it, even after a first reminder.
Compared with a simple letter, a complex one cut the response rate by an estimated 6.1 percentage points. Adding an information flyer cut it by 3.6 percentage points. Yet in the same experiment, making the possible benefit stand out more raised the response rate.
The useful distinction is between adding material and making the important thing easier to spot. More detail sometimes made the letter work less well. Pointing at the benefit helped people respond. Both the information and the way it was presented changed, so the result doesn’t tell us that shorter documents always work better.
Why it matters: Here the outcome being measured was whether people answered a letter. That matters when the point of information is to help someone take the next step.
Read the field experiment on tax-benefit mailings.
Randomized field experiment: a comparison you can use can change the choice.
Sometimes the missing piece is information laid out around your own case. In 2012, Jeffrey R. Kling and colleagues studied how older patients in Wisconsin chose their drug plans. Some were picked at random to get a comparison made just for them. The others got material that pointed them to where the information was.
The letters covered the 2006 enrollment period and plans for 2007. The researchers assigned 451 people to the groups. Their main analysis used the 406 people who finished the follow-up.
Personal comparisons raised plan switching from 17% to 28%. Predicted yearly costs were about $100 lower per person. These were predicted costs, not cash savings the researchers saw. The people in it were a selected group, and some dropped out before the follow-up.
Why it matters: The personal letter put the differences that mattered side by side for each person. Having information out there is not the same as having a comparison you can use. That helps explain why one more broad review may be less useful than settling one comparison you still can’t make.
Read the field experiment on personalized drug-plan comparisons.
Systematic review: well-organized information can improve what people know.
What people understand matters too. In 2024, Stacey and colleagues published a Cochrane review. It pulled together randomized trials of decision aids for adults facing a choice about health treatment or screening. The trials compared the aids with usual care.
These aids make the decision clear, explain the options with their benefits and harms, and help people work out what matters to them. Across the studies that measured knowledge, decision aids improved scores by 11.90 points on a 100-point scale compared with usual care.
Information built around a real choice can improve what people know about it. These aids also help people think the choice through. They do more than add facts. Usual care differed from study to study, and what improved was knowledge about these health decisions. So it doesn’t predict how much an everyday comparison sheet would help.
Why it matters: For an everyday choice, keep the details that show how the options differ. A simpler comparison is useful when it makes those differences easier to see.
Read the review of health-treatment and screening decision aids.
Randomized experiment: a faster decision still needs its own quality check.
A simpler document can save time without clearly improving the choice. John Beshears and colleagues studied this in a paper dated July 2, 2010, using the documents that explain investment funds.
Harvard staff, not faculty, chose how to split pretend investment money among funds. Their pay for taking part depended on how the funds really did. The researchers left out many people, mostly because of errors in the study materials. That left 186 people in the analysis.
People who got the short documents took 22.5 minutes to make their two choices. People who got the full ones took 31.2 minutes. The choices they made were not significantly better in the ways the researchers measured.
Why it matters: Saving time is useful. It doesn’t show that people chose better, and the statistical result doesn’t prove the choices were equally good. These are two separate questions: did the information make the task easier, and did it help the person choose well?
Read the manuscript on simplified investment disclosures.
Laboratory experiment: useful information still has a price.
Even information that could improve a judgment may cost more than it’s worth. In 2016, Mordechai Z. Juni, Todd M. Gureckis and Laurence T. Maloney made this trade-off explicit.
Twenty-eight graduate students, postdoctoral researchers and lab managers tried to find a hidden target on a touchscreen. Extra dots gave clues to where it was, but each dot cut the reward for hitting it.
People asked for a different number of dots when the costs and rewards changed. In one condition, they collected more dots, on average, than the number expected to give the best reward. In the other, their average wasn’t significantly different from that number.
Why it matters: In this task, the odds and the rewards were set, so the researchers could work out the best balance. For an everyday decision, you’ll have to judge those costs and benefits yourself. The useful lesson is to weigh what another check could tell you against what it costs in time, money or delay.
Read the experiment on information sampling and its costs.
What this means for you
Give each new search a job. It might answer a question that could change your choice, or make an important comparison clearer. Then weigh what that answer could change against the time, money or delay it costs.
Decide what the choice has to satisfy
Write down the action you’re choosing and when you need to choose it. Then separate must-haves from nice-to-haves. A must-have is a requirement the option has to meet. A nice-to-have helps you choose among options that meet the requirements.
In the fictional transcription-tool trial, your must-haves are language support, the export format you need and a price you can accept. A review praising a handy extra doesn’t tell you whether those are covered.
With your requirements written down, you can see what you still need to find out. You can also spot details that make an option more appealing without answering an essential question.
Look for an answer that could change your mind
Before you open another source, finish this sentence: If I learned ___, I might choose ___ instead.
Include evidence that could rule out your current favorite. In the fictional trial, learning that the export format you need isn’t available would change the choice. Another review repeating general praise wouldn’t settle that.
Ask what is behind your growing confidence. A new answer about a must-have gives you a reason you can name. Feeling reassured without a new reason is a prompt to look at the evidence again. It isn’t a sign that the checking is done.
Make a smaller working view without losing what matters
Put the relevant options side by side against the same criteria. For each one, record the evidence you have, anything important you don’t know, and where to find the full source.
Keep any condition that could change the choice. A summary might say an option meets your requirement, but only under a condition you haven’t checked. If that condition affects your decision, go back to the full source and check it.
The comparison should show you a gap, not hide it. Mark an unanswered question as unknown. Then look for a credible answer to that question instead of collecting more general descriptions.
A shorter summary can make this work easier. But as the investment-document experiment shows, ease and speed are not the same as the quality of the choice you end up with.
Challenge the choice before you stop
Ask for the strongest realistic reason your preferred option might be wrong. Check credible evidence for that possibility. If sources disagree on something that could change your choice, settle that disagreement before you rely on the ones that agree.
In the fictional tool choice, a summary might say export is available while the full source adds a condition. Check whether that condition applies to your trial. If you can’t settle it, keep it marked as unknown and make it your next targeted check.
Now think about one more affordable check. Could its answer realistically change the decision enough to justify the time, money or delay? A quick check on an open must-have is worth something different from one more broad review that repeats known points.
All of these have to be true before you stop optional searching. Your must-haves are covered. Nothing still open could affect safety. No affordable next check has a real chance of changing the decision that is worth its time, money or delay.
Then decide, or start a trial you can reverse. If an important unknown remains, make the next search specific or get qualified help. A time limit can remind you to review your progress. Reaching it doesn’t answer an open requirement.
Keep a reason to reopen the decision
Write down what new fact, result or changed condition would make you think again. In the fictional tool trial, a new privacy requirement would reopen the comparison, because it changes what an acceptable option has to satisfy.
For choices you make again and again, keep your reasons and what you expected next to what really happened. Look at whether the information you gathered helped with the question you faced.
Don’t judge your approach from one outcome alone. A choice can turn out well through luck, or things can change after you decide. Note those uncertainties when you decide what to repeat next time.
How about your daily life?
Where are you still reading without a clear question for the next source? You might be comparing tools for a trial you can end, or choosing an everyday purchase you can return.
You could also have the opposite problem: plenty of material, but no clear comparison on a requirement that matters. Think of a situation where you’d like to understand what is still unknown.
Write down a possibility you’d like to explore. One suitable idea is enough; add another if it helps you choose.
| A situation in my daily life | What I still want to understand |
|---|---|
| ____________________ | ____________________ |
| ____________________ | ____________________ |
My chosen situation: ____________________
Choose a specific decision you can reverse, with no unresolved requirement that could affect safety. Choosing it doesn’t mean you’re ready to act. Carry it into the checkpoint below; you can change your choice if it doesn’t fit.
Work through your decision
Start with your chosen situation. If you came straight here, choose a specific everyday decision you can reverse. Nothing about it should be unresolved if it could affect safety. If something is, get it checked properly or get qualified help before you use this stopping checkpoint.
- Name the decision and its requirements. Write the action and the deadline. Separate what the choice has to satisfy from features you would prefer.
Decision, deadline, essentials and preferences: ____________________ - Name an answer that could change your choice. Complete this sentence: If I learned ___, I might choose ___ instead. Include a fact that could rule out your favorite.
Decision-changing question: ____________________ - Build a comparison you can use. Put the relevant facts and open questions for your options beside the same essential criteria. Keep links to your sources and any conditions that could change the result. Mark missing information as unknown and look for a credible answer.
Options, evidence, conditions, sources and unresolved questions: ____________________ - Check the strongest realistic challenge. Name a credible reason your current choice could be wrong. Where the evidence conflicts on something that could change your choice, settle that before you rely on repeated agreement.
Challenge, evidence checked and what remains unresolved: ____________________ - Choose whether to stop or check further. Stop optional searching only when all three things are true. Your must-haves are covered. Nothing still open could affect safety. No affordable next check could really change the decision enough to be worth its time, money or delay.
If they all hold, decide or begin a trial you can reverse. If an important unknown remains, name the one check or the qualified help you need. Reaching a time limit you set yourself doesn’t settle that unknown.
Decision or trial, or the next specific check or help needed: ____________________ - Record a reason to reconsider. Name a new fact, result or changed condition that would reopen the choice. For decisions you make again and again, record what you expected and what happened. Don’t judge the process from one outcome alone.
Reopening trigger and, where relevant, expectations to review: ____________________
Ask Jing
Become a Jingius.
Your next step
You’ve finished when you can explain the choice and why further optional searching can stop, or when you can name the important unknown and your next check. Keep the reopening trigger with your decision record.
A note on the evidence
These studies measure different things: how sure people felt, how often they were right, who answered a letter, who switched plans, what people knew, how long a choice took and how much people searched. Their results help you tell those outcomes apart. They don’t set one limit on information for every decision.
The clinical review is about structured support for health choices. The sports and dot experiments used tasks set up in a lab. The insurance study had a selected sample, and some people dropped out. The investment study left out many people because of problems with the materials. Apply the lessons with those settings in mind.
How we prepared this module
This module links findings from experiments and one clinical review to questions about how we gather information day to day. A fictional transcription-tool choice shows the difference between repeated praise and a fact that changes the decision. The written checkpoint helps you apply that difference to one situation of your own.
Sources
- Claire I. Tsai, Joshua Klayman and Reid Hastie. (November 2008). Effects of amount of information on judgment accuracy and confidence. Organizational Behavior and Human Decision Processes, 107(2), 97–105. Online proof, pp. 2–3, Experiment 1 Method and Results; proof pagination differs from final journal pages.
- Saurabh Bhargava and Dayanand Manoli. (November 2015). Psychological Frictions and the Incomplete Take-Up of Social Benefits: Evidence from an IRS Field Experiment. American Economic Review, 105(11). Author-hosted typeset proof; Introduction, p. 3; Research Design II.A, p. 10; Table 5, p. 19, Response column (1).
- Jeffrey R. Kling, Sendhil Mullainathan, Eldar Shafir, Lee Vermeulen and Marian V. Wrobel. (2012). Comparison Friction: Experimental Evidence from Medicare Drug Plans. Quarterly Journal of Economics, 127(1), 199–235. Author manuscript; IV.D, Experimental Intervention, and VII, Discussion.
- Stacey D, Lewis KB, Smith M, Carley M, Volk R, Douglas EE, Pacheco-Brousseau L, Finderup J, Gunderson J, Barry MJ, Bennett CL, Bravo P, Steffensen K, Gogovor A, Graham ID, Kelly SE, Légaré F, Sondergaard H, Thomson R, Trenaman L and Trevena L. (29 January 2024). Decision aids for people facing health treatment or screening decisions. Cochrane Database of Systematic Reviews, Issue 1, CD001431. 2024 update; abstract, Selection criteria and Main results, paragraph beginning “Regarding attributes related to the decision-making process”.
- John Beshears, James J. Choi, David Laibson and Brigitte C. Madrian. (July 2, 2010). How Does Simplified Disclosure Affect Individuals’ Mutual Fund Choices?. Author-linked manuscript; published as a chapter in David A. Wise (ed.), Explorations in the Economics of Aging, University of Chicago Press, 2011. Experimental Design, p. 6 and footnote 4; Results, pp. 11–13.
- Mordechai Z. Juni, Todd M. Gureckis and Laurence T. Maloney. (July 2016). Information sampling behavior with explicit sampling costs. Decision, 3(3), 147–168. Author-hosted in-press manuscript; Method, p. 12; Results 1.1–1.2, pp. 15–16.
