Say you’re thinking about a tool that suggests replies to customers. You picture the typing it could save. Then you think about letting it send those replies on its own.
The words might be the same, but your role has changed. In the first setup, you can fix a reply before it goes. In the second, a mistake may already have reached the customer.
That difference is easy to miss when you call both setups automating messages. The useful question is what the person and the tool will need to get done together.
This module is for ordinary work where automation can make mistakes, unusual cases matter, or putting things right takes effort.
What you’ll get from this module
In this module
By the end of this module, you’ll see more clearly the work that automation still leaves to you. We’ll use a design framework and research on the jobs people are left with. We’ll also use simulator findings, a workplace study and an official investigation. You’ll learn how to look at what you hand over, checking and putting things right. You’ll pick an everyday situation, and map a workflow using a sample you can throw away and a pretend problem.
Ask Jing
Become a Jingius.
What changes around the automated step?
Design framework: separate the kind of help from the authority you give it.
Raja Parasuraman, Thomas B. Sheridan and Christopher D. Wickens (2000) described four types of automation. A tool can gather information, make sense of it, choose what to do, or carry it out. Within each type, help can range from all by hand to fully automatic.
A system can mix these types at different levels. So calling a tool automated leaves an important question open: which part does it control?
The framework also looks at how far a tool can be trusted, and what happens when an action is wrong. Some tools that handle information can get it wrong. For these, it says to keep the original information within reach. People should also know the tool can be wrong.
Why it matters: This helps explain the opening example. Suggesting a reply and sending it carry different authority, even when both use the same text. Being able to see the original information also matters when you need to check a suggestion.
Read the framework for types and levels of automation.
Conceptual analysis: people need support for the work that remains.
Lisanne Bainbridge’s 1983 analysis looked at automation in industry, with examples from flight decks. She saw that designers could automate some tasks and leave people a mix of leftover jobs. Those jobs often had too little support.
Those jobs included watching whether the system was working properly, and getting help or taking over when it wasn’t. Watching and taking over are different jobs: spotting trouble is only the start of dealing with it.
Why it matters: So counting the steps a tool removes can miss an important part of the job. A person may do less routine work and still need information and skills for unusual cases. The lesson is to look at those leftover jobs, without assuming they cancel out every benefit.
Read the analysis of duties after automation.
Workplace study: assistance can produce a real benefit.
Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied AI help given to customer-support agents. Different agents got it at different times. Their November 2024 v2 manuscript covered one firm and the firms it hired to do its work.
Agents could edit or ignore the suggestions. For agents whose call quality was recorded, the study estimated 15% more problems solved per hour on average, compared with before.
The gains varied from worker to worker. The most skilled agents saw small drops in quality. The main rollout wasn’t assigned at random. So the estimate depends on how well the comparison separates the AI’s effect from other changes.
Why it matters: People still handled the conversations, and with AI help they solved more problems per hour. They could choose whether to use a suggestion. But the study didn’t test whether that freedom caused the improvement.
Read the versioned study of AI-assisted customer support.
Simulator experiment: the quality of the advice changes the result.
Nadine B. Sarter and Beth Schroeder (2001) put 27 airline pilots into three groups at random. The pilots flew pretend landings in icy weather in a simulator. One group had no aid. One had a display describing the ice on the plane. One had a display telling them what to do.
Accurate help improved how they handled the plane. With wrong advice, pilots did worse than those with no aid, especially when the display recommended an action. When they had to recover from a stall with wrong information, pilots shown the icing reacted more slowly. But they made fewer errors than those told what to do.
Why it matters: That trade-off matters: speed and correctness can point different ways. These were specialist simulator tasks. They don’t tell us how much an office tool would help or harm. They do explain why a smooth routine demonstration leaves a separate question about wrong advice and putting things right.
Read the experiment on icing decision support.
Combined research: more automation brings different benefits and costs.
Why it matters: Linda Onnasch and colleagues combined results from 18 experiments in a 2014 research review, called a meta-analysis. Across these experiments, more automation helped with routine work. But it made things worse when the automation failed. People were also less aware of what was going on. These costs showed up most when a tool moved from making sense of information to choosing the action.
Read the meta-analysis of automation and performance.
Simulator counterstudy: the expected pattern did not always appear.
Why it matters: Greg A. Jamieson and Gyrd Skraaning Jr.’s 2020 study tested licensed crews from nuclear power plants. They followed set steps in a full-scale simulator. With more automation, crews were more aware of what was going on. The expected changes in workload and performance during failures did not appear.
Read the nuclear-control simulator study.
Methodological reply: researchers disagreed about what that study could show.
Why it matters: Christopher D. Wickens and colleagues replied in 2020 that the nuclear study hadn’t properly tested the trade-off. It hadn’t measured routine work. So it couldn’t weigh the gains in routine work against the costs of failure. They also argued that the study might not have been able to pick up some differences. These were the concerns of the reply’s authors, not an agreed answer to the disagreement.
Read the methodological reply.
Multitask experiment: reliability mattered, but two automation assignments showed no detected difference.
Why it matters: In a separate multitask simulation published in 2015, Onnasch compared automation that gathered information with automation that carried out an action. Reliability affected people’s attention and performance. She found no effect from giving the automation one of those two parts of the task rather than the other.
Read the multitask study of allocation and reliability.
Taken together, these findings don’t give us a simple ranking of automation stages. Naming the stage tells you what the tool does. It doesn’t tell you how well the whole task will go. You still need to look at routine work, mistakes and how people and tools work together.
Official investigation: a message needs a response path.
The U.S. Securities and Exchange Commission’s 2013 account of Knight Capital’s trading incident described failures in how software was put live and controlled. It reported 97 automatic error emails on the day. Nobody acted on them.
Those emails weren’t designed as system alerts. The SEC found several failures, including poor procedures for responding; Knight settled without admitting or denying the findings.
Why it matters: The useful difference is between information existing and someone being able to act on it. This was a complex trading incident, with several failures adding up. It shows why a warning needs someone to get it. They need enough context to use it, and a response they can actually carry out.
Read the official account of Knight Capital’s control failures.
What this means for you
Look at the whole workflow you plan to use. A step that is removed can be valuable, and the work that is left still needs to make sense. You can look at both without assuming automation will either solve everything or create more work overall.
Name the delegation precisely
Say whether the tool gathers information, makes sense of it, recommends a choice or carries out an action. Then say whether it offers help or acts on its own. One workflow may contain several of these setups.
In the made-up reply example, collecting messages is different from summing them up. Suggesting a reply is different again from sending it. Naming those differences tells you where checking and authority need attention.
Don’t rank the stages from safest to most dangerous. Filtering information can hide something important, while a limited automatic action may be easy to check and undo. Look at what would follow, and what you could see, in your real situation.
Make the remaining work visible
List what someone still needs to check: missing inputs, wrong summaries, poor recommendations or unexpected results. Name who will do that work and what information they need.
For a suggested reply, you may need the original message and the policy that applies to judge whether the answer fits. If you can see only the suggestion, write down the missing information, rather than treating your approval as real checking.
Keeping information within reach gives checking something to work with. It still takes attention and effort. Count that effort when you compare the new way with how you work now.
Separate permission from a practical chance to intervene
Name who can approve, correct or stop an action. Then ask when they can do it. Being allowed to correct a reply after it has gone is different from having a chance to stop it being sent.
Look at what happens next. Does the output change a record, set off another action or create a promise? A mistake in a copy of a table you can throw away is different from one that has already reached a customer.
When a mistake spreads or is hard to undo, think harder about how to put it right. Find out what context, access and skill the responsible person would need. If an important part is unknown, find it out before you rely on the setup.
Trace one exception from detection to recovery
Choose a likely unusual case, such as an unclear instruction in a sample message. Describe what would show the problem, who would notice it and how much time they would have to respond.
Carry on through the response. What would the person need to understand, correct or stop? Would the original information still be there? A notification only helps if the planned response can actually be carried out in that situation.
Use a sample you can throw away, and pretend the problem happens without real effects. Never create a failure in live customer, trading, aviation or other systems where it would matter, for this exercise. If you can’t keep the test contained, write the situation through on paper and mark the uncertain behavior as unchecked.
Compare completed work, including correction
Watch routine work and the pretend problem separately. Write down what the tool removed, what was left for people, and what putting things right took. Compare the total effort of finishing and correcting, alongside the quality of the output.
A quick suggestion that needs a lot of checking may save less than it seems to. Count the time spent finding missing information and fixing the result before you judge the saving. A useful suggestion that is easy to check may still save effort. Compare the finished result as well as the effort. Finishing faster is less useful if the answer no longer does the job. Let what you see explain the difference, rather than assuming either result.
A small trial can show a missing step. It can’t show how often rare failures will happen. Finish with a clear description of the setup, what you saw and what is still open. No single score decides for you whether to adopt it.
How about your daily life?
Where are you thinking about handing over part of a task? You might be summing up messages, preparing suggested replies or formatting a copy of a table you can throw away.
Think about where the result goes and what you would still need to check. Choose something you can try with a sample you can throw away, rather than a real commitment.
| A situation I want to explore | What I hope the automation will improve |
|---|---|
| ____________________ | ____________________ |
| ____________________ | ____________________ |
One good idea is enough. Which would be most useful to look at now?
My chosen situation: ____________________
Carry it into the exercise below. You can change your choice if you can’t keep the test’s effects contained.
Map one automated workflow
Start with your chosen situation. If you came straight here, choose an ordinary task where mistakes, unusual cases or putting things right deserve attention. Use a sample you can throw away, and keep any pretend problem away from real effects.
Never create a failure in live customer, trading, aviation or other systems where it would matter, for this exercise. If a contained test isn’t possible, describe the steps on paper and write down what you can’t check.
- Describe the work and delegation. What should the finished task achieve? Name gathering, making sense, choosing and acting wherever they apply. Say what the tool does on its own and what it only suggests.
Task and delegation: ____________________ - Name the remaining duties and authority. List the checking someone must do, who does it and what information they need. Name who can approve, correct or stop an action, and whether they can step in before anything happens.
Checking, information and authority: ____________________ - Trace an exception and its consequences. Choose a likely unusual case. Write down what would show the problem, who would notice it and how long they would have to respond. Say what the person would need to take over or put things right: the information, the access and the skills. Check that taking over would actually be possible. Include any records, policies or later actions the mistake could affect.
Exception, detection and recovery path: ____________________ - Observe a contained example. Use the sample you can throw away for ordinary work, and a safely pretend problem. Write down what was removed, what was left and what correcting took. Compare the total effort of finishing and correcting with how you work now, alongside the quality of the result.
Observed benefit, remaining work and correction: ____________________
If you can’t test without real effects, write a walkthrough on paper and mark the behavior as unchecked. A small trial can’t show how often rare failures happen. - Keep the result and the unresolved question. Describe the workflow as you now understand it. If information, authority or a way to put things right is missing, name what you need to find out, and who can help, before you rely on it. Use what you saw without turning it into a single score for adopting or rejecting.
What I learned and what remains unresolved: ____________________
Ask Jing
Become a Jingius.
Your next step
You’re finished when you can describe the work handed over, the jobs people still have and the way to put things right—or say exactly where that understanding is still missing.
A note on the evidence
The specialist simulations look at certain people, tasks and systems. Their findings don’t rank automation stages for every case. The customer-support result is about help in one workplace setup. The investigation records failures that added up in a different setting.
Together, they help explain what to look at. Your own workflow still needs attention to its information, its effects and the ways to put things right.
How we prepared this module
This module links a design framework and a conceptual analysis with simulator research, a workplace study and an official investigation. Made-up reply and table examples turn these ideas into ordinary work. The written exercise helps you look at one planned workflow and write down what is still uncertain.
Sources
- Raja Parasuraman, Thomas B. Sheridan and Christopher D. Wickens. (May 2000). A Model for Types and Levels of Human Interaction with Automation. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 30(3), 286–297. Author-uploaded journal reproduction; abstract, §III p.288, §IV.B.1 pp.291–292 and §IV.B.2.
- Lisanne Bainbridge. (November 1983). Ironies of Automation. Automatica, 19(6), 775–779. Original journal scan; §1 and §1.1, p.775.
- Erik Brynjolfsson, Danielle Li and Lindsey Raymond. (7 November 2024 PDF dateline; v2 submitted 6 November 2024). Generative AI at Work. Author manuscript v2. Published in The Quarterly Journal of Economics, 140(2), 889–942, May 2025; publication date 4 February 2025. Findings here use v2: abstract, introduction p.2, §3.2 pp.13–14 and §4.1.
- Nadine B. Sarter and Beth Schroeder. (Winter 2001). Supporting Decision Making and Action Selection under Time Pressure and Uncertainty: The Case of In-Flight Icing. Human Factors, 43(4), 573–583. Author manuscript and publication abstract; Methods, Participants; Discussion opening and paragraph beginning “This study examined not only”.
- Linda Onnasch, Christopher D. Wickens, Huiyang Li and Dietrich Manzey. (Online 20 August 2013; issue May 2014). Human Performance Consequences of Stages and Levels of Automation: An Integrated Meta-Analysis. Human Factors, 56(3), 476–488. Abstract, Method, Results and Conclusion.
- Greg A. Jamieson and Gyrd Skraaning Jr.. (Online 26 July 2019; issue June 2020). The Absence of Degree of Automation Trade-Offs in Complex Work Settings. Human Factors, 62(4), 516–529. Abstract, Method and Results.
- Christopher D. Wickens, Linda Onnasch, Angelina Sebok and Dietrich Manzey. (Online 28 January 2020; issue June 2020). Absence of DOA Effect but No Proper Test of the Lumberjack Effect: A Reply to Jamieson and Skraaning (2019). Human Factors, 62(4), 530–534. Abstract, Approach and Results.
- Linda Onnasch. (Online December 2014; issue April 2015). Crossing the boundaries of automation—Function allocation and reliability. International Journal of Human-Computer Studies, 76, 12–21. Publication abstract and author manuscript reproduced as Study III, pp.82–105, in Onnasch’s doctoral thesis. The thesis labels this manuscript “under review”; it is not the final journal layout.
- U.S. Securities and Exchange Commission (SEC). (16 October 2013). SEC Charges Knight Capital With Violations of Market Access Rule. Press release 2013-222. Paragraphs beginning “According to the SEC’s order” and “The SEC’s order also finds”; response-controls item and settlement paragraph.
