Automation Complacency: Where Will the Thinking Go?

A navel-gazing reflection on GPT, human cognitive effort, and the stepladder to the future. Where do YOU stand?

Thanks to Dr. Brian Montague for prompting this post with his quote during a recent Large PIG meeting:

I find that I do a lot of my thinking when I write my progress note. If/when ChatGPT starts to write my note, when will I do that thinking?

That stopped me in my tracks.

We are so hell-bent on simplifying our work, reducing our EHR burden — we sometimes forget that this work is more than just pointing, clicking, and typing.

It is also about thinking. It is about assembling the data and carefully coaxing information and patterns out of our patients through skillful interview, parsimonious lab testing, and careful physical examination. It is how we, as physicians and APPs, use our bodies and minds to craft an image of the syndrome, the disease: our hidden opponent.

Just like inserting a PC into the exam room changes dynamics, inserting GPT assistants into the EHR causes us to rethink … everything.

Pause to reflect

First, I think we should recall the technology adoption curve.

I fully acknowledge that I am currently dancing on the very peak of the peak of over-inflated expectations. Yes. That’s me right at the top.

Of concern, viewing recent announcements from Google, Microsoft, and many others gives me chills (sometimes good, sometimes not) of what is coming: automated, deep-fake videos? Deep-fake images? Patients able to use GPT to write “more convincing” requests for benzodiazepines? Opiates? Other controlled meds?

And yet, think of the great things coming: GPT writing a first draft of the unending Patient Advice Requests coming to doctors. GPT writing a discharge summary based on events in a hospital stay. GPT gathering data relating to a particular disease process out of the terabytes of available data.

Where do we think physician/APP thinking might be impacted by excessive automation?

Automation Complacency

I refer you back to my book review of the book “The Glass Cage” by Nicholas Carr. Although it was written to critique the aircraft industry, I took it very personally as an attack on my whole career. I encourage you to read it.

In particular, I found the term “automation complacency” a fascinating and terrifying concept: that a user who benefits from automation will start to attribute more skill to the automation tool than it actually possesses, a complacency of “don’t worry, I’m sure the automation will catch me if I make a mistake.”

We have already seen this among our clinicians, one of whom complained: “Why didn’t you warn me about the interaction between birth control pills and muscle relaxants? I expected the system to warn me of all relevant interactions. My patient had an adverse reaction because you did not warn me.”

Now, we have this problem. We have for years been turning off and reducing the number of interaction alerts we show to prescribers precisely because of alert fatigue. And now, we have complaints that, “I want what I want when I want it. And you don’t have it right.” Seems like an impossible task. It is an impossible task.

Thank you to all my fellow informaticists out there trying to make it right.

GPT and automation: helping or making worse?

Inserting a Large Language Model like GPT that understands nothing, but just seems really fluent and sounding like an expert, could be helpful, but could also lull us into worse “automation complacency.” Even though we are supposed to (for now) read everything the GPT engine drafts, and we take full ownership of the output, how long will that last? Even today, I admit, as do most docs, that I use Dragon speech recognition and don’t read the output as carefully as I might.

Debating the steps in clinician thinking

So, here is where Dr. Montague and I had a discussion. We both believe it is true that a thoughtful, effective physician/APP will, after interviewing the patient and examining them, sit with the (formerly paper) chart, inhale all the relevant data, assemble it in their head. In the old days, we would suffer paper cuts and inky fingertips in this process of flipping pages. Now we just get carpal tunnel and dry eyes from the clicking, scrolling, scanning, and typing.

And then, once we’ve hunted and gathered the data, we slowly, carefully write an H/P or SOAP note (ok, an APSO-formatted SOAP note). It will include the Subjective (including a timeline of events), Objective (including relevant exam, lab findings), Assessment (assembly of symptoms into syndromes or diseases) and Plan (next steps to take).

During this laborious note-writing, we often come up with new ideas, new linkages, new insights. It is this piece we worry most about. If GPT can automate many of these pieces, where will the thinking go? I do not trust that GPT is truly thinking. I worry that the physician will instead stop thinking.

Then there is no thinking.

Is this a race-to-the-bottom, or a competition to see who can speed us up so much that we are no longer healers, just fast documenters, since we are so burned out?

Who will we be?

Radio vs TV vs Internet

My optimistic thought is this. Instead of GPT coming to take our jobs, I’m hopeful GPT becomes a useful assistant, sorting through the chaff, sorting and highlighting the useful information in a data-rich, information-poor chart.

Just like the radio industry feared that TV would put them out of business (they didn’t), and TV feared that the Internet would put them out of business (they didn’t), the same, I think, goes for physicians, established healthcare teams, and GPT-automation tools.

Lines will be drawn (with luck, we will draw them), and our jobs will change substantially. Just like emergent (unpredictable) properties like “GPT hallucinations” have arisen, we must re-invent our work as unexpected curves arise while deploying our new assistants.

A possible stepladder

I believe physician thinking really occurs at the assembly of the Assessment and Plan. And that the early days of GPT assistance will begin in the Subjective and Objective sections of the note. GPT could, for example, do the following:

SIMPLE

• Subjective: Assemble a patient’s full chart on-demand for a new physician/APP meeting a patient in clinic, or on admission to hospital, focusing on previous events in can find in the local EHR or across an HIE network, into an easily digestible timeline. Include a progression of symptoms, past history, past medications.
• Objective: Filter a patient’s chart data to assemble a disease-specific timeline and summary: “show me all medications, test results, symptoms related to chest infection in the past year.”
• Then leave the assessment and planning to physician/APP assembly and un-assisted writing. This would leave clinician thinking largely untouched.

MODERATE

• Subjective and Objective: GPT could take the entire chart and propose major diseases and syndromes it detects by pattern matching and assemble a brief page summary with supporting evidence and timeline, with citations.
• Assessment and Plan: Suggest a prioritized list of Problems, severity, current state of treatment, suggested next treatments, based on a patient’s previous treatments and experience, as well as national best practices and guidelines. Leave the details, treatment adjustments and counseling to physicians/APPs interacting with the patient. Like Google Bard, GPT may suggest ‘top 3 suggestions with citations from literature or citations from EHR aggregate data’ and have the physician choose.

DREAMY/SCARY

• Subjective and Objective: GPT could take the Moderate tools, add detection and surveillance for emerging diseases not yet described (the next Covid? The next Ebola? New-drug-associated-myocarditis? Tryptophan eosinophilia-myalgia syndrome, not seen since 1989?) for public health monitoring. Step into the scanner for full body photography, CT, MRI, PET, with a comprehensive assessment in 1 simple step.
• Assessment and Plan: GPT diagnoses common and also rare diseases via memorizing 1000’s clinical pathways and best-practice algorithms. GPT initiates treatment plans, needing just physician/APP cosignature.
• A/P: Empowered by Eliza– like tools for empathy, takes on counseling the patient, discovering what conversational techniques engender the most patient behavior change. Recent studies already indicate that GPT can be considered more empathetic than doctors responding to online medical queries.

CMIO’s take? First things first. While we can wring our hands about “training our replacements,” there is lots yet to do and discover about our newest assistants. Shall we go on, eyes open?

This piece was written by CT Lin, MD, CMIO, UCHealth and Professor of Medicine, University of Colorado. To view his blog, Undiscovered Country, click here.