Say a tool writes your weekly update almost instantly. Then you spend the time you saved checking its claims and filling in missing details. You also move the result to where your team needs it.
The draft arrived sooner. But did the finished update take less work?
That question is the productivity paradox in everyday work. Better technology doesn’t always give you better results for the effort you put in. To judge the benefit, look at the finished work and everything it took to produce it.
This module is for people and small teams who do repeatable work. You should be able to judge its finished output and quality.
What you’ll get from this module
In this module
By the end of this module, you’ll know how to compare how fast a tool seems with the useful work it helps you finish. We’ll draw on history, tests with writers and coders, a study of customer support and a model of how firms invest in new tools. You’ll learn what to count. Then you’ll pick a situation from daily life and write a short comparison. It will help you decide whether to keep, change or stop.
Ask Jing
Become a Jingius.
Why a better tool can produce different results
Writing experiment: a clearly defined task can be a good place to start.
In a 2023 account, Shakked Noy and Whitney Zhang describe an online experiment with college-educated professionals. Each person did two writing tasks linked to their job. Between the tasks, some people were picked at random to get an introduction to ChatGPT. The rest did a different activity using Overleaf, an online document editor.
People introduced to ChatGPT could use it on the second task. Compared with the other group, they spent 40% less time on that task. Their average grades from professional reviewers went up by 18%.
These were made-up work assignments. As Zach Winn’s MIT report explains, they didn’t need exact facts. They didn’t need knowledge of a particular company or customer either.
The result shows that a suitable writing task can improve without redesigning a whole organization. Still, people got an introduction to the tool, and they changed how they did the task.
Why it matters: The useful difference is between producing a promising draft and meeting everything the real job demands. Any checking the test left out still belongs in your own comparison.
Read the authors’ account of the writing experiment. Read the MIT report on the study’s work requirements.
History: a new capability can make a different way of working worthwhile.
In a 1990 analysis, Paul A. David describes how American factories switched to electric power. At first, with “group drive”, electric motors each powered a group of machines through shared spinning shafts. That already used less power and gave better control over machine speed.
Later, “unit drive” gave each machine its own motor. That opened up more options for laying out the factory and moving materials through it. So changing the power system could also change how the work was organized.
This is a historical account with several influences at play. It isn’t an experiment that isolates the effect of redesign. Its lesson is that the way things are set up can hold back a new capability. That can happen even when simply swapping it in already helps.
Why it matters: So the writing experiment and the factory history answer different questions. One defined use may pay off straight away. Bigger gains may need changes to the work around it. Neither account says you must redesign everything before you try anything.
Read the historical analysis of electric power and factory organization.
Workplace study: an average gain can hide very different experiences.
Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied an AI assistant used in customer support. Their author manuscript is dated November 2024. It looks at data from 5,172 agents at a software company and the outside firms working for it.
The data run from September 2019 to June 2021. Agents got access at different times. The researchers looked at how things changed for agents after they got access. They compared that with agents who didn’t have it yet. This method is called difference-in-differences.
They reported that agents resolved 15% more issues per hour on average. That figure only covered agents whose records included the information needed to judge service quality. Less-experienced and lower-skilled workers gained the most.
The most experienced and highest-skilled workers got only a little faster, and their quality dropped slightly. Agents also did introductory training before they got access. The study didn’t separate the effect of that training from the rest of the rollout.
Access wasn’t handed out at random across the whole workforce. The findings apply to this service operation: its workers, its tasks and the way the assistant was introduced.
Why it matters: An average improvement can hide a poor fit for some people. Counting resolved issues also ties speed to a finished service result, rather than just counting messages sent.
Read the customer-support study’s author manuscript.
Developer experiment: feeling faster and finishing faster can disagree.
Joel Becker and colleagues at Model Evaluation & Threat Research (METR) studied 16 experienced open-source developers. They worked on 246 real tasks in projects they knew well. Each task was randomly assigned to allow or not allow early-2025 AI tools.
When AI was allowed, tasks took 19% longer to finish. Yet afterward, the developers estimated that AI had cut their time by 20%.
The gap is between the measured time and what the developers believed afterward, in this small, selected group. It’s a concrete reason to check the result instead of relying only on how productive the work felt.
The date matters. In a February 2026 follow-up, METR said bigger speedups were plausible. But the newer estimates were unreliable. There were problems with who took part, which tasks were included and how time was measured.
Why it matters: So the earlier slowdown isn’t a verdict on every coding tool today. The lasting question is how your own sense of things compares with the work you can actually measure.
Read the early-2025 developer experiment. Read the February 2026 follow-up on measurement difficulties.
Economic model: the cost of investing can come before the benefits show.
Erik Brynjolfsson, Daniel Rock and Chad Syverson built the “productivity J-curve” model in 2021. It looks at the investments made alongside new technology. These can be poorly measured in national accounts, the records used to measure an economy.
The investment may mean preparing people and changing how work is done, rather than buying physical equipment. That makes it harder to see what has been built for future work.
In the model, missing this investment can make early productivity growth look too low. Later, when the investment starts adding to output, growth can look too high, because that contribution isn’t properly counted.
Here is the useful distinction for you. Getting ready to do the work and doing the work again and again both use up resources. In your own comparison, keep setup and training visible next to the effort each job takes.
Why it matters: The model doesn’t prove that a particular use that is losing will pay off in the end. Count the investment so you can judge whether the result is worth it. The fact that such investment exists isn’t a reason to keep extending every trial.
Read the productivity J-curve model’s abstract.
What this means for you
Judge one defined use by two things: the finished work you can accept, and the total effort it takes. Start with the task and watch what happens. Change the work around it when you can see what is holding the result back.
Decide what counts as finished before measuring speed
Pick a task you do often. Describe what one usable, finished piece of it looks like. For a weekly update, that might mean the needed facts are checked. The update is ready for the people who will read it.
Hold both methods to the same quality standard. An unfinished draft and an accepted update aren’t the same unit, even if each fills one document.
Before you test, write down the smallest improvement that would make the tool worth adopting in your situation. Include the time, money and quality requirements that matter to the decision.
This stops a faster in-between step from quietly becoming the definition of success. It also gives you a reason to turn down a saving that is too small to be worth the extra cost.
Count all the work needed to finish
Record everyone’s time. Include preparing, doing, checking, correcting and handing off the work. Keep the time from start to delivery separate from the hours people actually work. Waiting for a reply can delay delivery without anyone working during the wait.
Make sure the units are comparable and meet the same quality standard. Then divide the number of accepted units by the total hours worked on the regular work. Count rejected work separately, but include the hours spent on it in the total.
Say both methods produce eight accepted units. Your current method takes two hours of work, so eight divided by two gives four accepted units per hour.
With the tool, doing the work takes one hour and reviewing and reworking it takes another. Eight divided by those two hours still gives four accepted units per hour, before setup costs.
The step you can see is faster, but the whole job hasn’t saved any work. If a targeted change cuts the total to 1.5 hours at the same quality, eight divided by 1.5 is about 5.33 units per hour.
These made-up numbers show why it matters which hours you count. Include all the effort needed for the accepted result, not only the step the tool speeds up.
Compare similar work and keep setup visible
Use tasks of a similar type and difficulty, including the usual hard cases. Where you can, split comparable tasks between the two methods at random. In a small before-and-after trial, extra practice or a different mix of tasks can change the result.
Give the tool the information the task needs, and learn how to use it. Record setup, training and money spent separately from the regular work. Then pick an amount of work and calculate a result that includes its setup and training time.
Stretch the made-up example to four batches of eight accepted units. At 1.5 hours of work per batch, the regular work adds up to six hours. Add one invented hour of setup: 32 units divided by seven hours gives about 4.57 units per hour.
Your current method would take eight hours for those four batches, giving four units per hour. Setup shrinks the advantage you saw when you compared only the regular work. Even so, in this example the tool still saves some work.
The amount of work you choose matters. The more work the setup supports, the more thinly its cost is spread. Look at money spent, delivery time and other limits separately. Units per hour of work can’t capture every reason to adopt or reject a tool.
Address the problem you found, then decide
When a result disappoints, find where the time or quality is going. Is the task a poor fit, is information missing, or are duplicate work, coordination and corrections eating up the saving?
Any change you propose should tackle what you saw. If missing information causes repeated corrections, improve that input and test again. If an unnecessary duplicate step eats time, first check whether you need it.
Keep a use when its gain is still worth it. That means after quality checks and the costs of putting it into practice. Maybe a result looks promising but isn’t settled. Then test one specific change, as long as the likely benefit is worth the cost of testing.
Stop if no realistic improvement is worth the cost of testing. Drop a use that’s losing; a historical example isn’t a reason to keep extending it.
If the comparison is uncertain, write down what you still need to know. Look again when the task, the tool or the quality requirements change in a way that could affect the result. An honest “not settled yet” is more useful than treating one good batch as a reliable gain.
How about your daily life?
Where do you regularly turn inputs into something another person can use? Think about routine updates, writing assignments or regular admin work with a result you can judge.
Which tool feels helpful, and which part of the finished job still takes effort? Write down a few possibilities, or just one situation that fits.
| A recurring situation | What I want to understand or improve |
|---|---|
| ____________________ | ____________________ |
| ____________________ | ____________________ |
Which would be most useful to look at now?
My chosen situation: ____________________
Choose work whose finished output and quality you can judge. Take that situation into the exercise. Choosing it doesn’t prove that a tool suits it.
Check one use of a tool
Start with your chosen situation. If you’re starting here, pick repeatable work whose finished output and quality you can judge. Compare your current method with one clearly defined use of a tool.
- Define the result. Write what counts as an accepted unit and the quality standard both methods must meet. Before you test, note the smallest improvement that would be worth the costs for your decision.
Task, accepted unit, quality and worthwhile improvement: ____________________ - Record the current method. Use comparable tasks, including the usual hard cases. Count accepted and rejected units separately. Add up everyone’s time for preparing, doing, checking, correcting and handing off, including effort on rejected work.
Accepted units / rejected units / total labor hours: ____________________
Elapsed turnaround and quality observations: ____________________ - Try the defined use. Give the tool the information the task needs, and learn how to use it. Keep task difficulty and the standard for acceptance comparable. Where you can, split comparable tasks at random. Record the same output, hours, turnaround and quality details for the method that uses the tool.
Tool method’s results: ____________________
Separate setup, training time and direct spending: ____________________ - Compare the complete effort. Divide the accepted, comparable units by the total hours worked on the regular work. For an all-in comparison, choose an amount of work. Add its setup and training hours to its regular hours before dividing.
Look at money spent, delivery time and quality alongside the result. If units, quality or task difficulty aren’t comparable, note the gap before you draw a conclusion. Small before-and-after trials can also reflect practice or changes in the work.
Recurring comparison / chosen workload / all-in comparison / limits: ____________________ - Keep, change or stop. Keep a gain that is still worth it after quality checks and the costs of putting the tool into practice. If a result is promising but not settled, name the problem you saw and one change worth the cost of testing.
Stop if no realistic improvement is worth the cost of testing, and drop a use that’s losing. If you don’t have enough evidence, write down what’s missing. Look again when the task, the tool or the quality requirements change in a way that could affect the result.
Decision, evidence, unresolved question or next action: ____________________
Ask Jing
Become a Jingius.
Your next step
You’ve finished when you have a local decision you can back up, or a clear account of why the result is still unsettled. Keep the comparison with its quality requirements and costs, so your conclusion stays tied to what you tested.
A note on the evidence
These sources measure different things. They cover graded writing, resolved support issues, coding time and modeled productivity growth. Their percentages can’t be added together or used as a forecast for your work.
History helps explain what’s possible; it doesn’t set a timetable for adopting anything. The developer results concern tools from a particular date, and the support findings vary from worker to worker. Keep the task and the actual comparison attached to any result you use.
How we prepared this module
This module links history, experiments, workplace evidence and an economic model to a practical comparison. That comparison looks at finished output and effort. The numbers in the worked example are made up. The written exercise helps you examine one use in your own setting.
Sources
- Shakked Noy; Whitney Zhang. (2023-06-07). The productivity effects of generative artificial intelligence. CEPR / VoxEU. Author account; “Experimental design” and “Results,” paragraph beginning “Consequently”.
- Zach Winn. (2023-07-14). Study finds ChatGPT boosts worker productivity for some writing tasks. MIT News Office / MIT Economics. Institutional report; opening task-limit paragraph and “Simulating work for chatbots”.
- Paul A. David. (1990-05). The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox. American Economic Review, Papers and Proceedings 80(2), 355–361. Published analysis; pp.357–358, group-drive and unit-drive discussion.
- Erik Brynjolfsson; Danielle Li; Lindsey Raymond. (Journal 2025-05). Generative AI at Work. Quarterly Journal of Economics, 140(2), 889–942. Author manuscript arXiv:2304.11771v2, 2024-11-06; Abstract, §§3.1–3.3, §4.1 and Appendix A.1.
- Joel Becker; Nate Rush; Beth Barnes; David Rein. (2025-07). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. Model Evaluation & Threat Research (METR). Research report; Abstract, p.1, and §2.
- Joel Becker; Nate Rush; Tom Cunningham; David Rein; Khalid Mahamud. (2026-02-24). We are Changing our Developer Productivity Experiment Design. METR. Methodological follow-up; opening paragraphs beginning “Unfortunately” and “Based on conversations”.
- Erik Brynjolfsson; Daniel Rock; Chad Syverson. (2021-01). The Productivity J-Curve: How Intangibles Complement General Purpose Technologies. American Economic Journal: Macroeconomics, 13(1), 333–372. Abstract, first paragraph.
