Clinical Warnings and the "Cry Wolf" Syndrome

My former software vendor employer had a great little business going. Like most clinical software companies, we resold another company's medication-related clinical decision support data to our customers, customized and tweaked to work with our particular applications. Every one of our users signed up for this service, knowing full well they couldn't possibly keep their files updated with detailed information on thousands of drugs.

It was pure gravy for us because we paid the data vendor a flat rate, with no marginal cost per new installation. Our customers had a barely noticeable clause in their contracts that authorized us to remove all drug information if they didn't renew, which would leave them with a brainless clinical system. Needless to say, the money kept rolling in. Our otherwise clueless executives would high-five each other behind closed doors for their genius and foresight in creating this seemingly endless source of low-risk, low-effort revenue, marveling that the market was insensitive to increasing prices and lack of further investment.

Customers didn't gripe. The IS folks figured they'd bought our systems to improve patient safety and outcomes. Not displaying allergy warnings was not a desirable option. We would manage the process for less than the cost of a full-time pharmacist. In the grand scheme of things, this moderate expense gave what seemed to be a solid return on investment and inarguable benefit to patients.

Not really, unfortunately. We repeatedly saw the problems among our customers:

• The IS departments simply didn't apply the updates because they didn't know how or didn't have the time. We would get access their system for some other reason and notice that drug updates were a couple of years behind.
• Users customized our information but didn't pay attention to warnings about our new data being in conflict with theirs. They either lost their changes or failed to get important new information that we'd sent.
• Users tried to set up their own information and didn't do it right, either getting none of the warnings they were trying to create or issuing them inappropriately.
• The IS department simply turned all warnings off due to complaints from doctors and nurses. I was stunned that even our hayseed hospitals would do something as stupid as disabling the technology they were paying for to protect their patients.
• And the most concerning problem: we were issuing so many warnings that they were being universally ignored. The important ones were getting missed in the clutter and patients were being harmed. We'd get calls to dig into the files to see if a pharmacist had received a warning when entering a particular order that harmed a patient. They almost always had.

On the provider side, my vendor gripes have been:

• Some vendors offer systems that users can't be customized to local practice.
• Systems that don't warn the IT staff directly of technical problems (server, PC, or database) that prevent one or more classes of warnings from displaying at all (we did this more than once on the vendor side, where some programming mistake completely killed certain warnings until the next release. Oops! Of course, we didn't tip the customers off.)
• Applications that display messages in random order in a fixed-size window with a scroll bar. Users can't see them all without scrolling, so they just sail through with a single click or keystroke, never seeing them all. Or, those that bring up one-at-a-time warnings with no end in sight, encouraging users to press Enter a bunch of times to buffer their way into order completion ("Just take the order, dammit.")
• Data and system vendors that insist on displaying every known warning for fear of running afoul of FDA labeling or liability lawsuits (with similar arguments for not allowing users to customize them.)
• And the most troubling once again: watching important warnings getting lost in the clutter of meaningless ones. I've investigated adverse drug events that seem indefensible: why didn't the idiot doctor realize he was ordering a medication to which the patient was allergic or the incompetent pharmacist read the warning that he was about to cause severe harm with a massive overdose to a newborn, both of which had triggered stern warnings of impending disaster? Answer: the user wasn't stupid, we'd just overwhelmed them with too many warnings.

You would think that clinical users like physicians and pharmacists would be the most vocal advocates of these warnings. You would be wrong in most cases. They simply tolerate them at best, ignoring them most of the time. If the clinicians don't find warnings userful, why do we pay so much money to display them?

We aren't just issuing a lot of insignificant warnings, we're also using systems that aren't capable of detecting potentially harmful situations. This is perplexing. Medication-related decision support  has been around for 30 years. Why does ISMP's 2005 pharmacy computer survey shows little improvement since 1999? Examples:

• 71% of systems can't detect a methotrexate overdose because of the way it's dosed
• 68% can't warn of high serum potassium levels when a new potassium order is entered
• 55% don't provide warnings when a renally excreted drug is ordered for a renally compromised patient
• 71% of systems routinely display meaningless warnings, desensitizing users to important ones.

We can't really blame the data vendors in many cases. It's the system vendors who often don't utilize their data correctly, don't support some of the screening methods for which the customer  is paying the data vendor, and who don't provide well-documented procedures for maintaining data and applying updates correctly.

Here's an interesting exercise. Contact your data vendor and find out all the screening their data supports. Then, compare it with what your system really does. Are you getting accurate dose checking for neonates? Drug-lactation and drug-pregnancy warnings? Lifetime dose checking of chemo? Total daily dosing warnings if a patient has orders for several products containing acetaminophen? Gender-specific warnings? Loading dose vs. maintenance dosing calculations?

Customers must accept some blame for the situation. Few of them fanatically monitor the warnings being issued and work with clinicians to fine-tune their data to reduce false alarms. That's assuming that their vendors provide the ability to turn off specific drug interactions, adjust acceptable dose ranges, and add dose checking for populations such as neonates and the elderly. For the IT folks, more warnings sounds like a good thing.

The Advisory Board Company's report, Hardwiring the Evidence: Customizing Systems to Drive Breakthrough Clinical Performance, cites studies in which over-alerting was one cause of an amazing 87% failure rate of clinical decision support to improve patient outcomes. We're spending a lot of money for technology that largely is doing nothing except annoying users.

I had a stroke of genius once about this situation. Each possible warning should be graded before being shown to the physician or pharmacist. How big an overdose is it? How serious is the drug interaction? Then, display only the two most serious warnings that score above a certain threshold. A therapy decision won't be changed based on the third- or fourth-most-important warning (and yes, some drug orders kick out five or more cautions.) The hard work is figuring out the scoring algorithm. Then, allow screening rules to be customized in new ways: for a specific physician, by patient location (ICU vs. med-surg,) or with a particular diagnosis.

I could go on, but I'll leave you with this challenge if you're a provider. Run a list of warnings that CPOE or pharmacy system users receive in a one-week period. How many warnings were issued per new order on average? How many of them led to the order being entered as originally intended, meaning the warning wasn't helpful? Then, run the ISMP test on your own CPOE or pharmacy system. Are you getting your money's worth, or just a false sense of security?

If you're a vendor, here's your challenge: provide a "did you find this warning useful" button on every message and log the result so the customer can chisel away mercilessly at meaningless warnings through ongoing file maintenance. Work with your data vendor to find ways to improve your application's use of their data. Add another button to each warning that says "don't show me this warning again." If a clinician is smart enough to write or enter orders, he or she is smart enough to decide that a particular warning is not useful to them, no matter what the armchair clinicians at First Data Bank or Multum think.

You wouldn't tolerate a car whose "idiot lights" warn you constantly of being 1 milliliter short on oil while not letting you know that you're about to run out of gas. It's puzzling to me why hospitals don't pay this issue more attention than they do or why vendors don't differentiate themselves on more discriminating methods of warning users. Or, as I always gripe, move more toward "guidance" instead of "warning" as a metaphor of interacting with clinical users. Maybe everyone has moved on to other problems, figuring this one's solved. It isn't.