I hate nature. I am a city boy. Having lived the first 30 years of my life in Manhattan, I am most comfortable surrounded with skyscrapers, subways, yellow taxis, glass, and concrete. Pigeons and squirrels are wildlife in my life manual, and elevators are a common way of commuting.
These days I find myself living and working in rural communities of New Jersey. I have to contend with deer, rabbits, and chipmunks using my perennial garden as a salad bar, and trees, grass and pollen causing me to be on a perpetual intake of antihistamines (and what has to be an unhealthy dependence on Afrin).
I hate — I mean, really hate — nature. Give me air conditioning, high speed internet and streaming video and I am happy. But flora and fauna are just awful. I enjoy and can appreciate a scenic view but would prefer to experience it on a 4K ultra high def television as opposed to standing “outside” (horrible word) being assaulted by nature’s bounty.
I know rural NJ is nothing like the rural of America’s bread basket, but by northeast standards, the county in which I work is rural. It is made up of 437 square miles of hills and farmland, with a population of about 130,000 people. Located in the western part of the state, it stands as the wealthiest, and healthiest county in the state, but its low population density begets a less-than-robust telecommunication infrastructure. It was for this reason we decided to build our own DWDM self-healing Fiber Ring in order to enhance our ability to deliver most excellent healthcare to the community.
It was 1:15 p.m., and the voice on the other end of the phone said, “The telephone service in the S&B building is down. The guys are checking the switches and ports.” I replied, “Thank you. Text me with an update. I’ll be back in the office in 45 minutes.” I returned to the meeting, thinking that if someone made an unauthorized configuration change, I was going to be very irked. You see, the S&B building on my campus held several physician offices as well as the primary office of a large surgical group. We all know how understanding physicians are with technology, and I am sure I don’t have to expand to the reader how surgeons are renowned for their calm and patience. Clearly we would need a customer service intervention at the very least.
About 15 minutes later I get a text that stated, “We just lost data to the S&B Building.” Oh great, I sputtered, and excused myself from the meeting, explaining there was an issue.
When I got back to the IT Offices, my CTO reported that the core switches and ports all seem to be fine and he dispatched one of the network engineers to the S&B building and was waiting for a report. No sooner were the words out his mouth when our NOC board announced that the Tin building just lost connectivity. The Tin building, which housed our ambulatory practice team, was adjacent to the S&B building on campus, and the next building further away from the hospital. There were some shared expletives, some head scratching, and in almost a cartoon-like caricature, a perplexed look on my network team which would make the Wile E. Coyote look insightful. These were very talented people; hence, my confidence factor crashed to the floor with a resounding thud.
We had what we thought to be a very robust network with decentralized backups and a great deal of redundancy. Earlier in the year, we committed to building out a second path for fiber in order to enhance our connectivity. I asked if the crew installing the new fiber could have damaged our connection and was told that the team was not doing any work on campus today. Noting that the county was doing some road work in front of the hospital, I asked if the county could have damaged the fiber, but was assures our conduit was nowhere near the construction site.
The NOC board then turned red again as the Burst building went dark. This facility is adjacent to the TIN building and next in line furthest from the hospital. Clearly we were suffering from some type of Cascade failure, and just as we were absorbing this new detail, the primary clinical system came crashing down. More expletives, more head scratching, more puzzled looks. And the day had been going so well.
The help desk and IT clinical team were deployed to all our nursing units to help with downtime procedures, and our network team was running a series of test. I turned to my CTO and said, “It’s almost as if someone is cutting through our fiber. Let’s send someone out to physically walk the ‘fiber path’ and see if they can find something.” He replied, “Our fiber is in concrete conduit — that can’t be it.” So I said, “Let’s walk it anyway. Something is not right.” A few minutes later, My CTO’s phone binged. A text message from the team walking the fiber. “We found the problem,” it read. Then came a second bing, with a picture.
A groundhog?! You have got to be kidding me! There in a utility hole was a groundhog, happily munching through 36 strands of fiber! Lord almighty, I hate nature.
What we were able to deduce in the aftermath was that one of the subcontractors left the cover off a utility hole into which the groundhog fell. Unable to extricate itself from the 12-foot deep pit, it started gnawing its way through the fiber causing angst and our service disruption. Animal control came quickly and emancipated the rodent from its entrapment.
Once the utility pit was vermin-free, our cable contractor was able to effect repairs, but that was not simple. Moe, as one of my staff named the insidious creature, had eaten the fiber near to where a conduit went under a 5-story parking structure, making it impossible to repair on site. We ended up at 5 p.m. on a Tuesday afternoon replacing 600 feet of fiber, from Moe’s utility pit to another utility hole on the other side of the garage. Our high-availability clinical system had its secondary data storage in the S&B building, so when the fiber connecting it was consumed, data could not be consumed by the secondary storage array causing it to crash. All in all, it was 12 hours of downtime.
My team and I consider ourselves to be pretty creative, but it never occurred to plan for a rodent attack. Yes, we have gone back and reviewed our disaster plan and held a formal RCA with changes suggested to both technology and downtime process. The physicians and surgeons, of which I was originally concerned, accepted the outage with exceeding grace, saying that even groundhogs understand the value of fiber in their diet.
But honestly, would any of you consider that a CIO would require experience in wildlife management? We presented the results of the RCA to our Quality Board who was as astonished as we were, and my department received nodes for a job well done. At the close of report to leadership I found myself thinking an Anglican hymn which reads:
All things bright and beautiful
All creatures great and small
All things wise and wonderful
The Lord God made them all
Lord, you and I have to have a talk!