Concept

What a money study measured decides what its result can tell you.

Issue

A claim about understanding your finances may rest on evidence about confidence, app use or reported debt instead.

Solutions

Learn to tell those results apart, then compare a claim with its evidence.

Say you read a headline claiming that letting an app track your money makes you understand it less. You want to know whether the research backs that up.

Finding a study about money tracking is only the start. The important question is what the researchers measured. Opening an app is one thing. Feeling well informed is another. Being able to explain your money is a third.

This module looks at one kind of claim: that letting an app track your money makes you understand it less.

What you’ll get from this module

In this module

By the end of this module, you’ll know how to keep a money study’s result apart from a bigger claim. A survey, a study of a mobile app, two field experiments and a comparison of debt reports show why those differences matter. You’ll see what they mean in practice. Then you’ll pick a claim you’ve read and write a conclusion that sticks to what was measured.

What did the researchers measure?

Survey: feeling knowledgeable and answering questions are different measures.

Lind and colleagues’ 2020 survey of 2,063 Swedish adults asked questions about money ideas. It also asked people to rate their own knowledge. People said how they handled money, too.

Both knowledge measures were linked to how people said they handled money. That held even after allowing for other skills and traits. So people’s confidence mattered to the result. But the two measures still told the researchers different things.

Why it matters: The questions tested what people knew about money. The self-rating showed how knowledgeable they felt. Neither tested how well they understood their own accounts. A link with reported behaviour doesn’t turn a confidence rating into that missing test.

Read the survey of financial competence and confidence.

Study of a mobile-app release: easier access can have benefits.

Carlin, Olafsson and Pagel’s January 2022 manuscript looks at the launch of a mobile app in Iceland. The service pulled people’s money details together in one place. Users could already see automatic records on a computer. The service showed information; it didn’t make payments or move money.

After the launch, the researchers found lower fees for payments that bounced. These are charges for trying to pay more than you have in your account or overdraft.

The result points to a gain from easier access. That holds only if nothing else that changed at the time explains it. The researchers didn’t measure understanding. Nor did they compare app tracking with keeping records by hand.

Why it matters: So you can accept the fee benefit while leaving the understanding question open. Lower fees are useful in their own right. They don’t tell us whether users understood their money better, worse or just as well.

Read the January 2022 mobile-access manuscript.

Field experiment reported by its research team: more use needn’t mean different spending.

Irrational Labs’ 2020 report describes 9,035 Android users who agreed to take part. Each was put at random into one of three groups: spending details, one overall budget or budgets by category. The experiment ran for 13 weeks.

People given budgets checked the app more often, but the researchers found no clear difference in average spending between the groups. In other words, they saw a change in use without seeing a change in spending.

This is the research team’s own report. More people dropped out of some groups than others. And the report doesn’t say how many were in the final spending comparison. Those gaps limit how confidently we can read the comparison.

Why it matters: The study didn’t test understanding. Its spending result also doesn’t prove that budgeting never helps. Keep the two results apart. Then you can accept the extra use without taking it as proof of lower spending.

Read the field report on budgeting and engagement.

Field experiment offering cashbooks: keeping records by hand needs its own evidence.

Smits and Günther’s 2018 study offered cashbooks to people in small-loan borrowing groups in and around Kampala, Uganda. The books let them write down money coming in and going out. Researchers followed up after 13 months.

The offer came with training, help and payments for taking part. The comparison group got no cashbook offer, though some of them already kept records. This matters. The study compared the whole offer with no offer, not keeping records with doing nothing.

The study found no improvement in number skills, reported loan repayment, income or food spending. Payments for taking part could explain the difference in savings. The study was small, and it relied on what people remembered. Both made small effects hard to spot.

Why it matters: This wasn’t a comparison of manual and automatic tracking, or a direct test of understanding. So it doesn’t show that writing things down repairs understanding lost to automation. It also doesn’t show that keeping records by hand can never help.

Read the published study of supported financial diaries.

What did each study measure?

Report comparing debt records: a total isn’t a personal score.

Brown and colleagues’ October 2013 report compared U.S. debt figures from household surveys with lenders’ credit records. Credit-card debt reported in the surveys added up to 60–63% of the amount lenders reported. The researchers compared patterns across the two sets of data. They didn’t match each home’s answers to its own credit records.

That percentage compares amounts of debt. It isn’t the share of people who got their debts wrong. Nor is it a score for how well one person knew an account. To say something about how accurate individuals are, we’d need evidence about individuals.

Why it matters: Differences in who and what each set of data covered also affect the comparison. The gap in reporting is worth looking at, but it doesn’t show that software caused it.

Read the report comparing borrower and lender debt records.

Fit the claim to the study

What this means for you

These studies answer useful but different questions. Together they don’t show that automatic tracking harms people by weakening their understanding. Nor do they show that keeping records by hand repairs such a loss. That doesn’t prove automation is harmless everywhere.

When you read a claim, put what it promises next to what the researchers actually measured. That helps you see which part the study supports and which part is still unanswered.

Keep the outcome’s name attached to the result

Start with what the researchers counted, asked or compared. Was it a confidence rating, answers to questions about money ideas, fees, app use, spending or reported debt?

Keep that wording in your conclusion. Swapping “app use” for “understanding” changes the claim, even if the rest of the sentence stays the same. If the study didn’t test understanding, leave that question open.

Say a made-up headline reads, “Budgeting apps restore financial understanding.” You compare it with the Irrational Labs report above. The researchers measured use and spending; neither tells you whether people’s understanding improved.

You could write: people given budgets checked more often, but the study found no clear difference in average spending. Understanding wasn’t tested, and different dropout rates limit the comparison. This study doesn’t support the headline’s promise of restored understanding.

Preserve who was compared and how

The comparison tells you what changed. Giving people a phone app for records they already had is not the same as swapping hand-kept records for an app. Offering cashbooks with support is different from asking everyone to keep records alone.

Write down the actual groups or conditions. Then ask whether the claim describes that same comparison. If it doesn’t, name the missing comparison rather than treating the study as a test of it.

Keep the kind of evidence clear, too. A survey can show that two things go together without showing that one caused the other. And not finding a difference doesn’t prove two approaches work equally well. Small effects can be hard to spot, as the cashbook study explains.

Check whether the result concerns people or totals

When you see a percentage, ask what it compares. In the debt report, the total credit-card debt reported in surveys was compared with the total lenders reported. The 60–63% figure describes that relationship between amounts.

It doesn’t tell you how many people understood their accounts. Turn it into a claim about people’s understanding, and you change what the number stands for.

Let missing information remain a useful result

If you can’t find what was measured or compared, write down what’s missing. Look for it in the source’s methods and results. If the information isn’t there, leave the claim unresolved.

You’re judging what the evidence supports. You don’t need to test your memory of your money, change an account or start a tracking routine. A clearly stated open question is a fine place to stop.

Which money claims do you meet?

How about your daily life?

Think of a piece about a money app, a social post about budgeting or a conversation about whether keeping records makes people better informed. Where would you like to tell the claim apart from its evidence?

You could pick a headline that interests you or go back to one of this module’s research accounts. One good idea is enough.

A claim or reading situationWhat I want to understand
________________________________________
________________________________________

Which would be most useful to explore now?

My chosen situation: ____________________

You can change your choice. Take this situation into the exercise below.

Compare a claim with its evidence

Start with your chosen situation. If you came straight here, pick a claim about automatic money tracking and understanding, or use the made-up headline above. You need the claim and its research account, not private account details.

  1. Write the claim. Keep what it promises clear, such as understanding your money situation.
    Claim: ____________________
  2. Write the measured outcome beside it. Name what was counted, asked or compared. If you can’t find it, note that gap and leave the claim unresolved.
    Measured outcome or missing information: ____________________
  3. Keep the study’s comparison and limits. Name who was studied and under what conditions. Keep clear whether the study found a link, tested a change or failed to find a difference. Keep a total apart from a score for one person.
    Comparison and important limits: ____________________
  4. Write the conclusion the comparison permits. Keep the name of what was measured. If the claim adds understanding, cause and effect or a personal fix that goes beyond the evidence, say what remains unanswered.
    Supported conclusion: ____________________
    Unanswered question, if any: ____________________

Ask Jing

Become a Jingius.

Check one claim against its study

Loading interactive tool…

Your next step

Finish when your note separates the reported result from the bigger claim, or clearly names the missing information. Keep this as a judgement about the evidence; it isn’t a test of your finances.

A note on the evidence

The studies looked at different people, made different comparisons and measured different things. Lower fees, a spending difference that wasn’t found and a gap between debt totals can all be informative without answering the same question.

Together, the research doesn’t show that automation weakens understanding in a way that harms people, or that keeping records by hand repairs it. That limit doesn’t prove such a problem can never exist.

How we prepared this module

This module brings together research on money knowledge, access, keeping records, app use and debt reports. It uses a made-up headline and a written comparison exercise to explain why conclusions need to match what researchers measured.

Sources