James Wellman, CIO, Comanche County Memorial Hospital
When James Wellman arrived at Comanche County, he was met with a thorough to-do list. The data center and network needed an overhaul, the current EHR system was fading into the sunset, and the hospital had “lost faith” in the IT department. But instead of running away, Wellman rolled up his sleeves and got to work. His first priority was to move IT away from “band-aid patching mode” and provide some much-needed stability, and then it was time to make some changes, which included transitioning to McKesson Paragon, and introducing a new approach to vendor partnerships. In this interview, he talks about selling to vendors like EMC and Brocade (instead of the other way around), what he learned from the job he didn’t get, why it’s wise to line up with large health systems rather than try to compete, and how to make changes without changing everything.
Chapter 2
- Breaking out of break/fix mode
- Transparency with the board — “This is where you are. Here’s what we need to do.”
- Challenging vendor parameters with EMC
- “The costs are the costs; we need to manage how we do this”
- Data & network overhaul
- Partnering with Brocade — “We sold them on the concept.”
LISTEN NOW USING THE PLAYER BELOW OR CLICK HERE TO SUBSCRIBE TO OUR iTUNES PODCAST FEED
Podcast: Play in new window | Download (Duration: 21:08 — 19.4MB)
Subscribe: Apple Podcasts | Google Podcasts | Spotify | Android | Pandora | iHeartRadio | Stitcher | Podchaser | Podcast Index | Email | TuneIn | RSS
Bold Statements
The hospital itself had lost faith in the IT department. They were continually under scrutiny. Systems were going down weekly. The stability was not there and they had gotten stuck into a band-aid patching mode as opposed to fixing the entire system.
I was very fortunate to come in to an administration that was open to that and said, ‘We’re looking for somebody with some external experience to come in here, make a difference and show us what we need to do, and then make smart decisions financially.’
I sat down and showed them, ‘this is where you are.’ Not doom and gloom and all is lost, but this is where you are. Here’s what’s contributing to our issues and our problems. Here are the things that we need to do and we need to change the way we approach this and the way we think.
The fact that they were very active participants and knew exactly where the money was going made it a little easier for them to do it. They weren’t any freer with the money, but they had an understanding.
We have a daily scrum to talk about these challenges and what are we going to do — what does today’s work hold for us, what are the challenges today, and what have we accomplished. We acknowledge each other. It’s become a very positive environment.
Gamble: Let’s talk a little bit about what needed to happen as far as cleaning up the data center, first when you arrived and you were looking at the whole situation with Horizon and Paragon, and then what needed to happen with the data center.
Wellman: Well, when I first showed up, one of the things that comes in here is you’ve got to staff that had seen a lot of people coming out of the corner office — there had been kind of a revolving door for a few years. So from their perspective, here’s another person coming in. It was a really good team, but they had lost a lot of the support they had. The hospital itself had lost faith in the IT department. They were continually under scrutiny. Systems were going down weekly. The stability was not there and they had gotten stuck into a band-aid patching mode as opposed to fixing the entire system. They weren’t being cured. They were just treating symptoms.
It’s really easy for that to happen when you stop and say okay, let’s figure out exactly what we have and what’s going on here. It was rather daunting; just the first-run physical issues when I came in here — looking at all of the network closets or IDFs, whatever we want to call them these days, the challenges that were there from security to no continuity to no standardization, code that was all over the place, the different switches and different locales, and no type of plan or long-term plan to get everything cleaned up. I think 75 percent of the data center servers were out of warranty and not necessarily on any type of coverage. So it was a Herculean effort with the local staff just to keep everything running.
The upside — and I think it’s true when you take a new job — when they’re interviewing you, you’re interviewing them. I literally wanted to find a place that 1) needed some help, because I wanted to be able to come in and make a difference, but 2) also needed to understand that and be willing to do what they needed to do to make those changes. I was very fortunate to come in to an administration that was open to that and said, ‘We’re looking for somebody with some external experience to come in here, make a difference and show us what we need to do, and then make smart decisions financially.’ Because obviously we just can’t come in and fix everything in a year. It’s just not fiscally responsible for an organization our size.
So we literally sat down and did some of the initial things that we needed to do. I had actually come up from the infrastructure side in my career. So this spoke to me a little more so than some small application issues or something that we couldn’t quite get in coding. We sat down and started looking in all the systems, stabilizing what we could, what was crashing, what was going on, what seemed to be the key problems, and made a few changes — not anything wholesale because we weren’t quite where we were sitting, and I did not know the depth and breadth of the issues that we had.
I reached out to several different companies to come in and do a comprehensive assessment of the entire physical infrastructure and systems in the organization, and finding somebody completely neutral was a bit of a challenge. Fortunately, I had met some people out of California at my previous position at the University of Kentucky. We had been building some brand new buildings — in fact, a 500-bed hospital — and I was intimately involved in that and I got this opportunity to meet a few people and I came across a company that wasn’t selling hardware or software. They were selling their expertise, and they had been around some really major healthcare organizations. I convinced them to come out to Lawton/Fort Sill, Oklahoma to see a mid-major hospital. We’re a large hospital on the smaller end of large, but cumulatively across the country, I think we represent quite a large margin for hospitals of that type, and they agreed with my summation of that. They came out here and it was a really positive outcome. They came in and together we were able to put together a portfolio of the entire hospital and everything that was going on and where exactly we stood.
From a technology perspective, anybody who’s been in the industry can sit down and say I want five 9’s in this beautiful data center and this perfect network infrastructure and all these brand new equipment totally running. But how do you get there? How do you convince a board — when a hospital this size is looking, do we buy a new CT scanner or do I spend it on IT? Part of that is changing the understanding that IT is healthcare in a hospital setting. We save lives. I tell my staff, you’re an intimate part of this. We help to make this a better place and we help provide healthcare. I’m not touching the patient directly, but I’m providing systems that allow the medical staff to make decisions quicker and get results faster, and therefore pass that on to the patients and the families, and that’s what we’re here for.
Gamble: What was the name of the company you dealt with?
Wellman: It is T2 Technology out of Los Angeles, California. The former CTO from UCLA Healthcare started that company and I had the opportunity to meet them, and they came out and actually did work for us at the University of Kentucky. Because it’s easy to go out and go to Cerner or McKesson and a lot of these other companies for that matter. If you want a network and go you to Cisco, they’re going to come in and tell you all the new Cisco products you need. Or if you go to Dell, they’re going to come in and tell you all the new Dell components that you need.
And that’s not to say that there are people who don’t do good jobs, but when I’m trying to earn the trust of my board and my administration, and I’m the new person here, I need to be able to look them in the eye and say, we need to replace it. Right now this is where we are. Here’s our network equipment. Here’s our data center equipment. Here’s our physical data center itself. Here are the challenges that we have and here are the things that we need to overcome, and here’s some of the costs. You’re looking at how do we have a comprehensive off-site back-up, how do we do disaster recovery, business continuity, and a lot of these things, from an IT perspective, had not been tackled at the level that they should have been. Because instead of doing that, everybody was spending most of their time doing break/fix work on a daily basis and no strategy and long-term planning.
And that’s one of the benefits of being the new guy. You get to just come in and say, ‘we’re going to make these changes. That’s one of things where you have to trust me.’ And with the support of the administration, again, we’re able to that. Without their support, I probably wouldn’t have lasted 18 months because people would have been frustrated with this, but we have to come in and tell them to be patient, that we’re going to make changes for the better and that has come to reality.
Gamble: I imagine that can be a tough sell, but like you said, a lot of the focus has been on just changing that perception of IT, and IT’s needs versus the hospital needs, and breaking all of that down.
Wellman: Yeah, and for years, you get a lot of people in the administrative part of a hospital looking at IT as a black hole. I think that’s a challenge no matter where you are. They’re saying, ‘we have to throw all of these money at you because of Meaningful Use and ICD-10 and HIPAA and all these new regulations that are coming out that make us put this in here.’ Years and years ago, when I first started, a lot of the IT stuff was a bit of a luxury. It wasn’t a mandated component. It made things more efficient, but now, you can’t operate a hospital without it. It’s pretty tough. You are essentially a cost center. People come in, they turn on the computer, and they expect it to work. They turn on or flip a light on or light switch on the wall, they want the lights to come on, and they expect those systems to be there 24/7 whenever they’re treating patients. To put that together and provide that these days can be a bit of a challenge on smaller organization; it’s a whole daunting task in a large organization. They each have their different nuances that they have. But like I said, for us, it’s one of finance and funding and narrow margins and you’re a not-for-profit hospital, so you’re trying to do what’s best for the community and for the organization.
Gamble: Now in terms of getting some board approval, was it something where, with the data center, you could say this is what we absolutely need out of our network and this is what you’re not getting. Was that part at least cut and dry?
Wellman: What I sat down and showed, ‘this is where you are.’ Not doom and gloom and all is lost, but, this is where you are. Here’s what’s contributing to our issues and our problems. Here are the things that we need to do and we need to change the way we approach this and the way we think. These are the items, and obviously we looked at average costs to correct these items and where we could spend money and work with the CFO quite closely to say, how are we’re going to approach this.
In our case, we operationalized several of the large cost components because things that other people take for granted were not available to us. A lot of people are doing remote hosting and for an IT guy, when people that come in who are not in IT say, ‘tell me about this cloud thing. We’ll just go to all of it in the cloud.’ It’s always an inside joke with IT people.
But our challenge specifically was that the actual network connectivity into the city of Lawton was very limited. You had numerous ISPs and groups that provide services, but they’re all sitting on one backbone. So it didn’t matter that I had 10 people that could bring me a network drop into the hospital; all of it’s going to the same vendor, and I’m in Tornado Alley. One event can completely bring me down. I have no redundancy.
And so we really started pushing a lot of other vendors to come in and make some changes and unique pathways so that we could take advantage of things like remote hosting. But in the interim we had to come up with some meaningful solutions, so we approached some large vendors like EMC and said, we have a deal for you. We want to use your product, but everywhere we go, everybody wants us to use cloud storage. Here’s our dilemma.’ So we proposed, why don’t you put your equipment in our data center and we’ll just pay you over this period of time based upon an aggregate utilization every 90 days? They said, well, that’s interesting. I guess that one went up the food chain for them, and then they came back to us and we met a few more times with some of their executives, and again, I sold my concept that smaller organizations look at us nationwide and this was a way to bring them into the fold.
They concurred and agreed with us, and so we became part of a reference site for them and started referencing to different banks and smaller hospitals about this process. We have state-of-the-art storage and digital backup equipment at our site and at our off-site location that we pay based upon utilization. For me to go out and buy all of that, while I’m buying everything else, was just not feasible capital option for us. Again, it was just a different way of approaching things. My CFO at that time looked at that and he said, ‘the costs are the costs; we need to manage how we do this.’ Part of this was changing perceptions. Previously IT who would just come in every year and throw a big number out there and say we need to replace all the PCs. Well, trying to replace 1,700 PCs every three years is not a good plan for anybody. It’s the same with the server base.
So we started breaking those things up. We’d stretch a few servers and applications, and we were very transparent with the board and with administration in saying, ‘here are the risks. Here’s what we’re going to do to mitigate those risks. Here’s what we will do if something goes down. Here’s why we made this decision or we want to make this decision.’ We presented that to them, and they were very active in selecting. In certain cases, they came back and said, ‘no, we don’t want any risk here. We want this segment moved up or this segment replaced concurrently with this one or upgraded concurrently.’ The fact that they were very active participants and knew exactly where the money was going made it a little easier for them to do it. They weren’t any freer with the money, but they had an understanding.
Meanwhile we were seeing results and stabilization of the system; a clean-up and a change in the organization and that becomes self-fulfilling prophecy in the organization, in my opinion. People aren’t running into the office and yelling if something is not going down. So they feel a little bit better. Their pride’s up. They’re going after things. They see an achievable goal and we work with them, and those things became reality. The whole mindset of the organization and the IT department changed.
We did everything from literally changing the lighting in the data center — when I came in it was kind of dark, and we brightened it up and got people active. We got them all together and we have a daily scrum to talk about these challenges and what are we going to do — what does today’s work hold for us, what are the challenges today, and what have we accomplished. We acknowledge each other. It’s become a very positive environment for that, but still a daunting one because, as we talked about in that data center, our physical floor was collapsing. So the joke around here was that computer crash was going to have a whole new meaning if we didn’t take care of that. But how do you replace a computer floor and a data center and not bring everything down? So that was an interesting project to undertake.
Meanwhile, there was no overhead cabling in a data center that should have had it 15 or 20 years ago. It didn’t exist; everything was under the floors. All of that came out in the assessment, along with a lot of photographs, a lot of pictures showing people the challenges of what we had to undertake — no cabling or labeling control in any of the closets. So for 20 years, it was people hooking something up and getting it done and walking away, and then when you had a problem, you’re trying to locate that cable and it was next to impossible.
Gamble: Yeah, I can imagine.
Wellman: So we started changing all that, but we had to eat the elephant a bite at a time. It was, go get this closet and clean it up. This is the one we’re going to fix. Okay, that one’s done. Let’s go get this one fixed.’ That’s how we approached it, and it took us two years to get all of that done, but we replaced every single network switch. We were very fortunate, again. We sold our story and our theory to Brocade Networks. They had just done a similar deal with Children’s Hospital of Los Angeles, and we said, come out to Lawton, Oklahoma, and do the same thing. And again, they kind of balked and said we don’t normally work in that area. We work a lot larger. But we sold them on that concept.
They have probably been one of the most supportive partners that we’ve worked with since then. They have just really embraced the concept. They’ve gotten behind it 100 percent. They send people here and work with us. It’s a continual design. We have, I feel, a true partnership, and they’ve been really active with this. With their headquarters out of Dallas, they’re three hours away from us, so part of our plan too is to offer to have people come in and have site visits. And of course we do reference sites, and we show them what we have, but stem to stern, a full Brocade shop. With the exception of my access point for wireless, everything in here is Brocade, and it’s been a marvelous transition in the network.
So we have a network that can rival anyone at this point from our perspective. That’s a very nice feeling to have for me because that helps me sleep at night. And they work with us — it’s a 24/7 support component. They work to train our staff, so like I said, they have sincerely embraced this project and this process, and that’s been a big reason for part of our success.
Share Your Thoughts
You must be logged in to post a comment.