Taking Command: Building A Network Operations Center

“I had no idea it was this complex,” exclaimed our corporate controller. “Do you think our clinicians understand how this works?”

This was her reaction to touring one of our tier-3 data centers, and no, I don’t think our clinicians understand the challenge and importance of data center management. I don’t want them to have to worry about it, but I would welcome their understanding of how it supports their work and the large expense.

Like many integrated delivery systems, we have grown through mergers and acquisitions. These additions have been strategic, for sure, but some have been turnaround situations as part of our mission to improve community health in eastern Massachusetts. For many years, these facilities continued to run on their legacy HIS platforms with local data centers. In recent years, a focus on information security, high-availability (HA) and disaster recovery (DR) has pushed us to consolidate data centers. In addition, one of our “corporate” data centers that originally served one of our academic medical centers desperately needed to be replaced.

Of course, investment in tier-3, security, HA and DR isn’t fully realized without a degree of real-time, automated monitoring. We’ve certainly had pockets of such monitoring across our infrastructure and application stack, but we knew it was time to “grow up” and build a network/security operations center or “command center.”

Although we still have much work to do, we have seen concrete steps toward these goals in this past year. We opened our command center in our Charlestown office and are mapping our monitoring tools for security, infrastructure and applications. The dashboards can be displayed at our Service Desk locations as well as our executive offices. Recently, we opened our new tier-3 data center and are 75 percent finished with our data center consolidation effort.

We have veteran infrastructure leaders who are acting like six-year-olds on Christmas morning — and they should be. The command center has been coveted for quite some time. We invested in our first tier-3 data center in 2005, moving from our space next to the morgue and medical waste incineration facility at one of our hospitals. Even before that move was complete, our data center leaders warned of the need to plan for replacement of the other one — the one near the Liquefied Natural Gas terminal in Boston Harbor with the leaky roof.

This need languished for years through changes in leadership and no fewer than three external consultant reviews suggesting the construction of a new facility; I own forcing the third review. I came into this role not convinced we should build a new facility — couldn’t we remediate the current one? Wouldn’t the cloud make this unnecessary? Couldn’t we co-locate for a period of time?

The Command Center had seemed like a luxury — until we grew to the point where it seemed silly that we didn’t have one for our $10B company. Our Director of Network Engineering and Telecommunications had the vision and had budgeted for the construction and initial “monitor of monitors” software. Through the right alignment of stars — available capital, business need and will — we constructed the room in six months. We had our eyes on a room that would work well, but then had to negotiate to relocate the people using it.

Perhaps most surreal is that once approved, these facilities seemed to spring up and come to completion with relative ease. We wonder how we lived as long as we did without them?!

In terms of our future direction, I’m optimistic. As some know, we are adopting an enterprise-wide clinical and revenue cycle system. This new system will improve our care delivery and billing processes, making it safer and more efficient for patients. We have two state-of-the-art data centers to host this system and plan to run production for six months in one, then six months in the other. We have a Command Center that also meets the most advanced standards to monitor our environment and reduce mean time to detection and mean time to repair.

I don’t think our clinicians understand all of this, but I don’t understand cardiac catheterization either. I just want to make sure our technology supports our clinicians to ensure the best outcomes for our patients.