Concept

A small tool decision can start with one real job and a clear next step.

Issue

An appealing demonstration doesn’t tell you whether the tool meets your needs once checking, cleanup and costs are counted.

Solutions

Understand what deserves a closer look, then make your first-pass decision.

Say you’ve opened a demonstration of a new formatting tool. The result looks tidy, and you’re already wondering what else it can do. But you haven’t decided what you need it to do.

Without that question, it is hard to know what the demonstration has shown you. A polished result could still change information you need to keep. A tool that takes longer to learn could also deserve a fairer test than its first few minutes.

You need to tell three results apart: the tool does the job, it fails something essential, or you still don’t know enough. Each calls for a different next step.

Use this first pass for small tool choices you can undo, whose main benefit you can test on a typical task. A big rollout, or a move you can’t undo, needs its own proper evaluation.

What you’ll get from this module

In this module

By the end of this module, you’ll have a clear way to decide whether to adopt a tool for one job, skip it or plan a limited second look. We’ll link research on choices, measured work and teaching customers with guidance about the cost of leaving later. You’ll look at what these findings mean, pick a situation from daily life and write down a practical next step.

What makes a first test informative?

Systematic review and meta-analysis: bring the alternative into view.

Allegra Maguire, Emil Persson and Gustav Tinghög’s 2023 review looked at experiments that made opportunity costs easier to notice. These are the benefits you give up when you choose one use of your resources over another.

Across 39 experiments, reminders about other uses changed people’s choices. The overall effect was small, and most experiments asked people to make imagined decisions.

Why it matters: The useful difference is between a price and an alternative. A price tells you what something costs. Naming another use of the money makes the choice between uses clear.

Read the review of opportunity-cost reminders.

The same idea gives you a question about time: what useful work would another tool trial take time away from? The research doesn’t tell you how much time that question might save.

Randomized field study: feeling faster and finishing faster can differ.

Joel Becker, Nate Rush, Beth Barnes and David Rein’s July 2025 paper studied experienced open-source developers working on software projects they knew. Sixteen developers finished 246 real tasks. Each task was randomly set to allow AI or not.

With AI allowed, tasks took 19% longer to finish. Afterward, developers estimated that AI had cut their time by 20%.

The developers thought AI had saved time, but the time recorded for their tasks showed the opposite. That gap explains why a pleasant experience isn’t enough to judge the result.

This was a small group using early-2025 tools, reported in METR’s research preprint. The result applies to that setting. It isn’t a current estimate for all AI use or everyday work.

Why it matters: In a February 2026 update, Becker and colleagues explained problems with who joined a later study, which tasks they sent in and how work time was recorded. Those data didn’t give a reliable new estimate to replace it.

Read the early-2025 developer study. Read the later measurement update.

For your own comparison, define what counts as finished, including the checking needed for a good enough result. How a tool feels to use matters, but so does the work needed to finish the job.

What does a first impression of a tool show?

Consumer experiments: an aid’s design changes what it helps with.

Nicholas H. Lurie and Na Wen’s 2014 paper looked at tools for comparing products across several features. Sorting puts options in order by one feature. Elimination removes options from view.

Across three experiments, the tools helped or hurt depending on the choice. Sorting with no limits improved the quality of decisions when features didn’t pull against each other much.

High conflict means choosing between options with competing strengths. In that case, sorting alone made decisions worse. Offering elimination alongside sorting helped reduce that problem.

Why it matters: Rearranging a list can make one feature easier to compare without settling the trade-offs between features. The aid needs to help with the choice you actually face.

Read the study of simple decision aids.

For a small tool choice, keep the essential requirement in view. An attractive feature can’t make up for failing the job you need done. The shopping experiments don’t give you a personal score for choosing tools.

Field experiment: a learning step can change early use.

German F. Retana, Chris Forman and D. J. Wu’s 2016 paper reports a 2011 experiment by a cloud-service provider. The service supplied computing resources over the internet. Of 2,673 customers who signed up, 366 were offered early guidance on basic features.

The published abstract reports that the offer halved the number who left in the first week and reduced their support questions that week. These early effects faded within a week, while total use of the service was higher over eight months.

Why it matters: Staying with a service, asking fewer questions and using it more are different from getting more benefit from it overall. The finding still shows why the help on offer at the start matters when you judge early use.

Read the customer-education field experiment.

For your own test, ask whether you got to the task at all. If you were still setting up, a useful next test may need a learning step first. If you finished the task and it failed a requirement, you have a clear failure to deal with.

Official guidance: weigh the benefit alongside the effort of leaving.

UK Government guidance on cloud lock-in advises organizations to weigh benefits against risks and to think about the cost of leaving. The version updated in September 2026 also explains that avoiding dependence can bring costs and compromises of its own.

Why it matters: This guidance is about organizations choosing cloud services.

Read the guidance on managing cloud lock-in.

For a smaller tool choice, it offers a useful question before you commit: what would leaving take?

A cost of leaving is something to weigh alongside the benefit. It isn’t an automatic reason to reject a useful service. Knowing how to cancel, take your work elsewhere and leave the service helps you understand the commitment.

Find out what leaving would take

What this means for you

Use the first pass to choose the next step. You can leave with a limited decision to adopt, a reason to skip or a clear question worth looking into.

Define the job before watching the demonstration

Write down the job, how you do it now and the least you would accept as a result. This gives the trial something concrete to answer. Hold both ways of working to the same standard for the output. Otherwise, a quicker or simpler result could look better because it leaves out part of the job.

In the made-up formatting example, say the requirement is a readable document that keeps all the information it needs. A tidy output that changes that information fails the requirement. Its looks don’t cancel the failure.

Choose a small task that stands for the work you really want help with. Count checking and cleanup when you compare the result with how you do it now. Write down whether the result is better, about the same, worse or still unknown, and say why.

Look at the benefit and what it requires

Ask what improves after checking and cleanup. Then ask whether it fits how you work and handles the formats you need. Think about setup, learning, and whether you can try a little before a bigger commitment.

Include cancelling, exporting your work and how much effort leaving would take. Mark missing information as unknown, instead of assuming the answer is good. If an essential fact is still unknown, wait before you commit. Decide whether another check is worth doing.

Finally, name another useful use of the time and money. Compare the trial with that. You may still choose the tool when its expected benefit matters more, but now you can say what you are giving up.

Keep separate answers to these questions. Don’t add them up into a score that lets several small advantages hide one essential failure.

Give an unknown a specific next test

If setup uses up the first pass, the result is unknown. You don’t have to force a live trial just to fill the page.

A second look needs a useful job the tool could plausibly do, and one open question. Explain why the benefit may take longer to judge. Then choose one learning or setup step that matters.

Plan to try the tool through a normal round of the work, with the same quality standard as how you do it now. Decide what evidence would change your mind. Set a limit on time or money, and a date to decide.

For example, a tool that needs setting up before it can handle a job you do often may be worth that limited look. A vague hope that you’ll find a use for it one day isn’t the same kind of reason.

On that date, adopt, skip or give a clear reason to look further. Don’t let the trial keep running without a decision. How long you need depends on the task. A provider’s trial deadline doesn’t tell you how long a fair test will take.

Make the decision, then stop evaluating

Adopt for the job you tested only when all the conditions hold. The output meets the standard you set first. The advantage, or another benefit you value, is believable. The known costs and limits are acceptable, and no essential fact is still open.

Skip when there’s no real job, an essential requirement fails or the expected benefit doesn’t justify the known costs. Write down the reason, so the decision has a clear basis.

Try again later when a clear useful job is plausible, but missing evidence, setup or learning stops you judging fairly. Write down the limited second look, rather than leaving another open-ended search.

Once you’ve written down your decision, stop testing. Reopen it when your needs change enough to matter, believable new evidence turns up, the method you chose fails or the planned review comes round. A feature announcement alone needn’t reopen it.

Where are you with the tool you have in mind?

How about your daily life?

Is there a tool you’re thinking about for formatting documents, organizing information or a work task you do often? What would you want it to do better than how you do it now?

Write a few possibilities if that helps you choose. One good idea is enough.

A tool choice or situationThe job I want help with
________________________________________
________________________________________

Choose one small decision you can undo to carry into the exercise. Choosing it doesn’t yet mean the tool deserves to be adopted.

My chosen situation: ____________________

Make your first-pass decision

Start with your chosen situation. If you came straight here, pick a small tool choice you can undo, and a typical task that could show its main benefit.

A big rollout, or a move you can’t undo, needs its own proper evaluation. This exercise can name its next step, but can’t approve the larger commitment.

Set aside ten minutes for your first pass. Use the times below to organize the session, and mark an unfinished result as unknown.

  1. First minute: define acceptable work. Before you open a demonstration, write down the job, how you do it now and the least you would accept. Commit to choosing the next step when the time is up.
    Job / current method / requirement: ____________________
  2. Next four minutes: try a representative task if feasible. Hold both ways of working to the same standard, and count checking and cleanup. Write down better, about the same, worse or unknown, with a reason. If setup stops you testing fairly, write unknown.
    Observed result and reason: ____________________
  3. Next three minutes: examine fit and costs. Write down the benefit, how it fits your work and formats, setup and learning, whether you can try a little, and what result you can look at. Include cancelling, exporting, commitments and how hard leaving would be. Mark missing answers as unknown.
    Benefits / constraints / costs / unknowns: ____________________
  4. Last two minutes: choose the next step. Name another useful use of the time or money. Adopt for this job only if the output meets the standard, the benefit is believable, and the known costs and limits are acceptable. If an essential fact is unknown, wait before you commit.
    Skip if there’s no real job, an essential failure or too little expected benefit. Try again later only for a clear, plausible use, with evidence that could change the decision. Don’t add up a score.
    Alternative use / adopt, skip or retry / reason: ____________________
  5. If retrying, bound the second look. Write down why the benefit may take longer to judge, one learning step, a normal round of the work and the same output standard. Add the evidence you need, a limit on time or money and a date to decide.
    Question / learning step / work cycle / requirement: ____________________
    Evidence / cap / review date: ____________________
    On that date, adopt, skip or clearly justify looking further. Don’t quietly let it run on.

Stop testing once you’ve written down the decision. Reopen it when your needs change enough to matter, believable new evidence turns up, the method you chose fails or the planned review comes round.

Ask Jing

Become a Jingius.

Give one tool a ten-minute first pass

Loading interactive tool…

Your next step

You’ve finished when you have a decision and a reason, or a clearly limited look at an open question.

A note on the evidence

The research looks at different results: mostly imagined choices, recorded work time for developers, the quality of shoppers’ decisions and use of a service. These results help explain different parts of a tool decision, and shouldn’t be treated as the same kind of benefit.

Keep the early-2025 developer result in its own setting. A useful comparison for you still needs your task, the output you would accept and the real costs.

How we prepared this module

This module links a research summary, field studies, shopping experiments and official guidance with a practical decision exercise. A made-up formatting example explains how an appealing result can fail a requirement.

Sources