Five Stages of Clinical Software Oversight: Do You Know Where Your Vendor Is?

A recent HIStalk poll asked readers, "Should clinical information systems be subject to oversight?"A surprising 75% of readers said yes, preferring (by a three-to-one margin) that such authority be given to an independent body rather than to the FDA. Recent reports of patient harm caused by poorly designed and implemented clinical information systems no doubt influenced the voting.

The FDA has been making quiet threats for at least 15 years to impose some sort of clinical IT system oversight, terrifying vendors with visions of red tape, expense, and loss of competitive edge. FDA's traditional domain has been biomedical devices, those electronic systems that replace the judgment of a clinician with built-in rules and tools. However, the more advanced CPOE and other clinical decision support systems become, the harder it is to tell where traditional FDA-regulated "devices" end and clinical IT begins. The odds of a patient's being exposed to a judgment-containing, non-FDA approved system (CPOE, medication management, clinical calculator, interfaced smart IV pumps, etc.) have never been higher.

Regulation is also a timely topic because, frankly, some vendors have done a horrid job in their own self-policing. Tales of incredibly broken code, dysfunctional applications, clinician gotchas, and dangerous design flaws are not uncommon as applications get bigger and smarter, products change corporate hands, and integration efforts push software well beyond the limits under which it was conceived.

I'm no software development expert, but as a clinician and businessperson I would expect the natural path of oversight to go something like this.

In Stage 1, vendors manage their own processes in whatever manner they choose, judged only by customers and prospects on whatever quality deliverables they can achieve. This is the ultimate free market. Prescriptive requirements and external review are not implemented. This Stage 1 is where most vendors are today, being semi-responsive to customers when it's expedient to do so, but without specific requirements or responsibilities to avoid patient harm by anything other than good intentions. Given the switching costs of clinical systems and the paucity of vendors, the free market may not work all that effectively, as you know if you've ever tried to pressure a vendor to make a programming change once you've signed the contract. You just paid $10 million without doing due diligence ... it's too late to start making demands and you don't really scare the vendor by threats of telling prospects about their indifference.

In Stage 2, individual vendors voluntarily choose to embrace quality management programs and certifications, such as ISO or Six Sigma. The goals of these programs are open communication with customers, a thorough definition of requirements, market validation of design, comprehensive product testing before release, notification and recall procedures for defects, and implementation of documented and repeatable processes throughout all of these phases. Theoretically, the benefits of this approach would reduce cost and improve quality, thereby offering vendors an incentive to move up from Stage 1. Vendors must not be convinced, however, since few of them in healthcare have bought seriously into these programs (although management lip service is not uncommon.)

In Stage 3, those standards from Stage 2 would be made mandatory for all clinical IT vendors. Spare bedroom programmers and college students couldn't just whip out software intended for clinician use. Minimum standards and documentation would ensure some base level of quality efforts. Oversight would be by independent third party, leveling the playing field for all and making sure that software is free of major deficiencies that could harm patients.

In Stage 4, FDA might mandate low-level regulation involving documentation, testing, and recall processes. Cost of compliance would be low, although vendors with ongoing quality problems would be subject to mandatory improvement. Only the smallest and poorest new vendors would find the process prohibitive.

In Stage 5, FDA would fully regulate clinical software as a medical device. Development cycles would elongate, costs would increase significantly, and vendors would be impossibly challenged to certify and stabilize an application built over a complex foundation of operating systems, OS patches, middleware, protocols, and interfaces. If this happens, expect many vendors to bail out and for those remaining to innovate even less than they do today. If you've ever dealt with an imaging vendor who won't even let you apply critical Microsoft security patches for fear of running afoul of the FDA, you know exactly what I mean.

I'm a free-market kind of guy, so I'm going to encourage our majority of vendors stuck in Stage 1 to seriously consider Stage 2. The risk, I believe, is that lack of adoption of that stage will leave no recourse except government intervention to force Stage 4 and maybe even Stage 5. You don't want that.

Building and maintaining clinical software and integrated clinical applications is a horrendously complex endeavor. I can't speak for all vendors, but here are my gripes about one of the main ones at our place:

• User suggestions are sometimes accepted, but with minimal commitment and delivery cycle measured in years. 
• Design is undertaken either in a vacuum or with one influential customers, often leading to a disconnect with what other users need.
• Development is compartmentalized by product even though they are now sold together as an integrated platform. The respective developers still don't talk to each other (and may not even be located in the same city or country.) They break each other's stuff constantly and the interaction between applications is poorly documented and underappreciated.
• The user interface is the absolute pits, ensuring that non-programming doctors and nurses will make mistakes.
• QA is apparently lax, given the plainly obviously broken GA code we get.
• Industry growth and turnover ensures that we deal with a lot of not-yet-competent vendor staff, greenhorns who don't even understand what the software already does, much less what it should do (it's not unusual that we have to explain to these young 'uns how their company's product works even though we're paying them.)
• The vendor doesn't consistently let us know when they find problems, even serious ones that might put our patients at risk for the months it takes to get a fix out. The motto of most vendors: don't tell anyone about a problem if it makes you look bad, if it requires your resources to correct the problem for the customer, or if competitors might find out.
• Ship dates are still king, with non-technical, non-clinical managerial types demanding that product go out the door no matter what its readiness to do so. Management doesn't usually get fired for quality; it's dates and costs that get them in trouble. Guess which one slips?

Now by rights, this vendor shouldn't be successful in selling new systems with these problems. That might be true in a world of perfect competition and universal knowledge, but in this reality one, they're doing OK. The market isn't pushing them from Stage 1 like it should and their competitors aren't much better. That's why I like a voluntary Stage 2, where vendors choose a quality program, report their progress in it, and hopefully compete more effectively as a result. We customers have to demand it, though: in our negotiations, in our contracts, when we're being asked to serve as a reference site, and when we write those large checks for tepid support. It won't get done otherwise.

If customers don't hold their vendor's feet to the fire and we can't get past Stage 1 within a year or two, then I say we go to Stage 3. Put together an independent body to certify the vendors and announce their results to the world with a KLAS-like grade card. Everything is easily quantifiable: how many defects you have, how many customers are satisfied with the design process, how you undertake testing, and how you handle user-reported problems. You want to kick that crap out the door so that Dirk Squarejaw gets his annual bonus? OK, but the industry's watching intently.

Given that I'm not necessarily in favor of government oversight, the free market seems to be the best driver. That means you, customers and prospects. Push your vendors to do better. And in fact, maybe individual implementations should themselves be certified, thus possibly preventing problems as experienced by Children's Pittsburgh through their own apparent rush to get Millennium running even with major workflow and software problems that eventually harmed patients. Would it be asking too much to have the frazzling, challenging work of both vendor and customer in building a system reviewed one more time before unleashing it on unsuspecting patients?

Think of HIT like medication errors. We need a non-punitive culture, a forum in which to openly discuss mistakes without blame, ongoing learning from site to site to prevent future mistakes, independent error reporting and review, working with customers to design systems with fewer mistakes, and a recognition that all of us could do a better job in reducing the chances that our work harms patients. The whole culture of safety thing. And vendors, listen to those clinicians who work for you when they tell you your software is dangerous to patients. They know.