What Does Cerner’s Outage Tell Us About The Cloud?

Dear HIS Pros,

After Cerner’s outage last week, I’m not sure if “Cloud Computing” is right for us. Can you break down this issue? How do we decide whether to process in-house versus remotely? What’s the impact on large versus small hospitals?

Vince Ciotti, Principal, HIS Professionals

Barry Mathis, Principal, HIS Professionals

HIS Pros response:

Let’s start with the semantics to make sure we are all disagreeing about the same thing J! For purposes of this response, we’ll lump together all of the following variants of remote processing: Cloud computing, remote hosting, co-locating, RHO, RCO, etc. There are many technical nuances among them, but they all boil down to the key issue of whether a hospital runs its own servers in-house, or lets a vendor run them at a remote location.

And we have to state Cerner has two of the best data centers in the country (we’ve audited many vendors’ data centers over the years), so what hit them this week could well hit any remote processing vendor and, indeed, over time, probably will! With that being said, as in so many HIT issues, there are both pros and cons, so let’s look at each of them:

PROS

Low Capital Costs – hospitals can save large up-front investments in server hardware and data center space by processing remotely.

Rapid Implementation – saving the time to order & install hardware and build out an old data center can save valuable time in earning stimulus funds.

Off-Load Security Risks – less time, effort and exposure building a complex firewall with intrusion detection and monitoring.

Greater Redundancy – far more resources in the form of virtual servers spread across locations.

Disaster Recovery – vendors have dozens or even hundreds of FTEs in its operations department, so more resources on call during outages.

Performance – most commercial data centers will guarantee response times of a few seconds, and the best even offer penalties to back up SLAs.

CONS

Higher Ongoing Costs – vendor profit margins mean that over 5 or 10 years, hosting fees are generally far more than in-house system depreciation. If you factor in the cost of annual increases and volume fluctuations (most RHO vendors limit your patient days, # of visits, MD users, etc.), ongoing fees are generally higher than in-house processing.

Rapid Implementation = Poor Implementation – the rapidity with which you can build a system via the cloud tends to short-circuit the customization and workflow review that makes for a good implementation. If the hardware is ready to roll in a few months and pre-loaded with canned screens and reports, then why spend so much time with building, testing, training, procedure-writing, down-time procedures, etc.

Transferred Security ≠ Liability! – most remote hosting vendors limit their liability in carefully crafted indemnification clauses that their lawyers spend more time crafting than their programmers do their software! The result: if (when?) things go wrong, you’re as on the hook as if it was your own servers or operators who caused the breach anyway… And if you think the cloud is more secure, just Google “security breach” and sit back for along afternoon of commercial vendor citations!

Redundant Redundancy – Although commercial vendors have far more servers for back-up, as their data centers grow into mega-proportion, the complexity of the entire process reaches a point where even their geniuses can take hours to find out just what went wrong, as at Cerner last week. A smaller & simpler data center is smaller & simpler to diagnose. And when it comes to balancing profit versus redundancy at a vendor, guess which wins?

Performance – Although the vendor may have stunning performance within their server suite, the communication lines between the hospital and the data center can vary drastically. So although the vendor may have unlimited scalability, your ISP may charge so much more for a connection that matches their processing speeds that the combination is unaffordable.

Hospital size is an interesting issue in this regard. We have consulted at hospitals across the entire spectrum of sizes:

–          Multi-hospital IDNs with many thousands of beds

–          Large hospitals and AMCs from 300 to 1,000 beds

–          Mid-size community hospitals from 100 to 300 beds

–          Small hospitals under 100 beds.

Interestingly, the smallest sites, especially Critical Access Hospitals (under 25 beds), are often in such remote locations with such tenuous communication lines, they almost always process in-house. Their costs for a few servers and a small room is minimal, so users of CPSI, HMS, Meditech and Healthland, the leaders in that space, almost always process in-house. Only newer vendors like NextGen and Prognosis are pushing the cloud there.

As you go up in scale through mid-size and large hospitals, the HIS vendor selected usually determines the processing route:

–          Mid-size systems like Meditech, QuadraMed, Keane, Siemens MedSeries4 and McKesson Paragon sites almost always process in-house, since they don’t offer any “RCO” options themselves.

–          High-end systems are split:

  • Siemens Soarian, Cerner Millennium and Allscripts (Eclipsys) sites almost always process remotely through their large (and highly profitable) data centers.
  • Epic, GE and McKesson Horizon sites almost always process in-house, although more and more commercial vendors like ACS and “consulting” firms like Xerox and Dell are hosting servers in competition with HIS and co-locating commercial vendors.

–          The largest chains like HCA, CHI, Tenet, etc., usually process at regional data centers themselves. They are well aware of the high profit margin of remote processing and would rather keep those dollars on their own balance sheet.

So there you have it: as complex an issue as there is in HIS circles, and one that needs a unique answer based on your hospital’s size, location and system. If it were simpler, then we consultants would all be out of work!

Share

Email Newsletter

Sign up to receive our latest updates delivered straight to your inbox.

Comments

  1. The key thing it tells me is that humans can screw-up anything. The outage was a human failure caused by a systems programmer messing with the Win active directory which would bring down any system whether cloud based or inhouse.

    The more important question is: Since such events are inevitable (can’t get rid of the humans.. and Murphy’s law) how much are you willing to spend to ‘guarantee’ a real time image backup with complete failover?

    Cerner obviously doesn’t have it, but then neither does any inhouse system I have ever seen. Time to bring back Tandem computers!!!

    Frank Poggio
    The Kelzon Group

    • Frank, glad it was you that responded. A man who knows what he is talking about! You hit the nail on he head. Even the tried and true Tandem might be in danger these days. True story, myself as the CIO not long after a go-live where I convinced a tough CEO and CFO to free up “extra” funds so that I might create a more fault tolerant system. A “NonStop” system complete with N+2 servers connected via a 10GB backbone synchronized in perfect harmony. All strategically located miles apart complete with RAID Technology, dual power, UPS the works. So, I ask you this riddle: What happens when a 3rd shift vendor technician connects at 3:30AM to delete “temp files” from my nice redundant SCO Unix Servers and forgets he is at the root > and uses a wildcard delete? You guessed it. Each UNIX server believing this to be a perfectly reasonable request from an ID with admin privilages proceeds to execute said command to the fullest. He removes every scrap of data from three perfectly synchronized servers with one keystroke. So it is the human that presents the never-ending challenge. By the way, it was a dedicated vendor that recognized the position they were in and chartered a Jet that flew a crew to my hospital within hours and had my backups restored and the gap between live and backup closed within 36 hours. Of course, they never would give me that guys name.

      Barry L Mathis
      H.I.S. Professionals

  2. Barry,
    Glad you liked my response…smart move you made, just hired the wrong turkey!
    I would guess you ran this system before full blown EMRs & CPOEs.
    What would have happened if you were down 36hrs with these apps in play? I bet the docs would have been lined up outside the CEOs office and screaming.
    What I find most interesting and ‘frightening’ is very few if any small/mid-sized hospitals have anywhere near that level of redundancy today…they can’t afford it. So I expect to hear more and more stories like this as time goes on. And it really doesn’t matter whether you are running on the Cloud or on the ground.
    Maybe we need redundant operators. One guy types /selects the console command, the other says “ARE YOU REALLY REALLY SURE?”…twice

Share Your Thoughts

To register, click here.