Q&A With Mount Sinai Senior Director of Cyber Tom Mustac: Probing Questions Can Help Operational Heads With BCP

How about this scenario: The technology that had allowed one nurse to safely keep watch over 20 patients at a centralized station is down, meaning a lot more clinicians are suddenly needed to cover the same load. What will you do? This is the type of emergency that Tomislav (Tom) Mustac, Mount Sinai Health System’s senior director and head of systems, cloud and biomed security, tries on a regular basis to get his organization to consider. In this interview with healthsystemCIO Founder and Editor-in-Chief Anthony Guerra, Mustac talks about how he encourages staff at tabletops to deliberately challenge each other. “So they’re going to think about when this happens, if it happens, how would I deal with it,” he says. “And it also helps us on the IT side to get resources dedicated to step up our technology, to step up our processes and improve.”

LISTEN HERE USING THE PLAYER BELOW OR SUBSCRIBE THROUGH YOUR FAVORITE PODCASTING SERVICE. 

Podcast: Play in new window | Download (Duration: 34:42 — 23.8MB)

Subscribe: Apple Podcasts | Google Podcasts | Spotify | Android | Pandora | iHeartRadio | Stitcher | Podchaser | Podcast Index | Email | TuneIn | RSS

Bold Statements

… the communication is vital so we actually talk through these things with each other, how we see things unfolding. Because as a technologist, I can come to you and say, “Well, we’re going to do X, Y and Z,” and think it’s all going to be great. But now when I have to involve others that actually have to work within that plan that I developed, they have a different scenario, or a different outlook, or they may not have the resources that I’m assuming they have.

But everything with technology seems to be a matter of time. When people say something can’t be broken into, it can’t be done, that’s dropping the gauntlet, and people start going out there and trying to do it. Eventually, somebody will find a crack in the armor somewhere.

… what I’ve always said to my friends on the manufacturer side is if you don’t do this, somebody else will, it’s a competitive world. So you’re going to see foreign competitors come in and start eating your lunch, because they’re going to come in with patchable operating systems to update security features.

Guerra: Tom, thanks for joining me.

Mustac: Good morning. Thank you for having me.

Guerra: Great, lots of good stuff to talk about today. Tom, can we start out with you giving me an overview of your organization and your role?

Mustac: Sure. So we are one of the major healthcare providers here in the New York City area, consisting of eight hospitals and well over 200 outpatient clinics and offices. We’re also a research and learning hospital. So a lot going on; a lot of diversity in the environment, both from a technology standpoint and the services offered. My role, I’m responsible for the cybersecurity of all our devices and helping to close down the many vulnerabilities and concerns that have been in the medical vertical for a number of years that are getting a lot of attention now.

Guerra: Certainly are getting a lot of attention. So, quite an interesting title, head of systems, cloud, and biomed security. Do you see those as different buckets that you have to cover? Or is it, in your mind, one thing?

Mustac: From a technology standpoint, there’s a lot of similarities because bits and bytes are bits and bytes, but from maturity levels of security, they’re different buckets. The traditional irons that we have: mainframes, desktops, laptops – all that stuff is pretty well covered and mature. The new stuff in our environments, obviously, is the big push for the cloud. And medical and IoT devices, I think, have grown in footprint tremendously over the past decade. And we didn’t even realize how much technology crawled into our lives.

Guerra: Very good. So you are responsible for the medical Internet of Things. Do you have other folks responsible for the non-medical connected devices?

Mustac: We do. I mean, we have a pretty broad team. And we try to divide up the work. We collaborate where we can and where there’s overlap. But we specialize in our areas to address the specific security concerns with specific device types.

Guerra: And we’ll get a little more into business continuity planning, but you’re going to have very specific business continuity planning for medical devices if they have issues and need to be taken offline. Is that an area that you also do a lot of work? Tell me about that.

Mustac: Yes. Well, with medical devices, it depends on the device type. I mean, with the bigger devices like an MRI or a CT, that’s big iron, you can’t move those when they stop working or if they’re out of commission, and you have to find a different location with different equipment. With the smaller devices, you can swap them out fairly easily and have redundancy, like your infusion pumps, patient monitors, things like that, you can swap that equipment out within a location. But the bigger devices, there is no swapping out an MRI, not in a day, certainly.

Guerra: Yes. You know, I’ve spoken to a CIO at a different health system. And she had this scenario in her head of having to take all the infusion pumps out of service due to a cyber incident. What are your thoughts around that scenario?

Mustac: Well, with infusion pumps, that’s always a fast moving component in a health system. I mean, there are tons of rental pumps that people use all the time because of surge capacity. We have multiple facilities, so we can borrow from other facilities and take 10% from each one to replenish and get some rentals. So there’s a lot of different approaches you can take. You can also eliminate, in some instances, I would expect the clinicians can go back to a normal drip bag and more supervision of that.

So there is no straight, simple answer for what I am going to do because the environment changes so quickly. You don’t know what’s going to be available for your use at that time. You don’t know what capacity is going to be across the system at each location. You don’t know what the rental company is going to have going on.

Let’s say a particular brand was hit by something major. And we all jumped to get those, and we all suck up that supply very quickly, then it’s first come first serve, because there’s not a full replacement for everything, everywhere.

Guerra: So, it’s almost like there’s only so much you can do, running out scenarios in your mind, and then we’re just going to have to figure it out. Because there’s a billion different scenarios. So it’s hard to have a playbook for each one, because there’s too many. Does that make sense?

Mustac: Yes, absolutely. And obviously, we also engage with our colleagues in the Office of Emergency Management. Every mature health system has a separate emergency management function that takes care of the overall enterprise and how we deal with an emergency. So they deal with the clinical side; what do we divert? What do we cancel, as far as procedures? So definitely, technology has a place at the table, and we discuss how we can support, how quickly we can come back, and all that. But ultimately, the clinical decisions are going to be on the clinical side, the system-wide function, we’ll see what our options are. What can we do? Can we just go into diversion or move patients around to help us with the situation.

So it’s a very complex, and I would say unnerving, situation for a health system to deal with because of all the complexity; because people don’t stop coming in the doors. I mean, if an emergency happens, and if it’s regional, it just keeps coming.

Guerra: So is it cybersecurity’s job to make sure Emergency Management understands the types of cyber incidents that can happen? And then they take it and run with it from there to a certain extent?

Mustac: Correct. I mean, there was an instance, I believe, about two years ago where a hospital in the US had to go back to paper. And when the day came, they didn’t have the paper. They didn’t think it through. So we have, in all these organizations, a lot of bright people with a lot of great experience. But the communication is vital so we actually talk through these things with each other, how we see things unfolding. Because as a technologist, I can come to you and say, “Well, we’re going to do X, Y and Z,” and think it’s all going to be great. But now when I have to involve others that actually have to work within that plan that I developed, they have a different scenario, or a different outlook, or they may not have the resources that I’m assuming they have.

So it’s really important that you do desktop planning, work through them all and really talk through the what ifs, and a lot of times when we go through these tabletops, I like to have a little fun with them and tell people, don’t be afraid to challenge each other. It’s not about embarrassing each other. It’s about surfacing all the different what ifs, making it a little bit uncomfortable. Meanwhile, let’s not just walk a straight path, because that happy path is not always going to happen when we need it. So let’s have those what-if scenarios, let’s challenge each other a little bit and get uncomfortable, and that’s how we learn and we evolve.

Guerra: Great points. Can you get into a little more detail about where cyber should lead and where it should support or collaborate?

Mustac: Yes. It’s important to keep the discussion going; never give up, keep the awareness going. Keep people aware of things that are happening in the news and how they’re relevant to your organization. It might be a similar structure or architecture that you’re using which is the same as this other organization that had an event. You might be using the same technologies.

And it’s important to demonstrate to people that this is real, I mean, we all like to watch a movie or some Netflix series in our downtime. And we see these scenarios play out. I think one of the most famous ones, there was one movie where – I forget what TV series it was – but the President’s pacemaker was attacked. And people look at that and say, “Well, yeah, that can’t happen. That’s all Hollywood, and it’s exaggerated.” When you play through some of these scenarios, people look at them and say, “Well, they’re not realistic, it can’t happen.” But if you really dig in the news and you’re focused on it, there’s a lot of stuff that’s happened to us over the past three, four years as a society that we always said can never happen.

I mean, if you look at the medical side, COVID I mean, who would have imagined that COVID would ever happen and we’d be locked in our homes. We never thought that people would be attacking hospitals from a cyber standpoint, but they’re doing it for various reasons. Sometimes we’re just a casualty, and they don’t know what they’re hitting. But other times we’re a direct target.

Guerra: Yes, I spoke to one CISO, who talked about doing some tabletops. And some of the people involved said, “This is ridiculous. This scenario is too outlandish.” And he said, “Well, this just happened two months ago at another health system.” Right?

Mustac: Yes. So it’s important to keep that understanding in front of people so that they know that place that had this happen, they were using technology X version seven, and we have that same thing, or maybe we’re even a version behind, you never know. So it’s important to keep people aware, let them know that this is real. We’re not exaggerating, and we can’t address all risks, but we can manage risk within the context of our environment and our resources. And you need to prioritize what’s going to give me the most bang for my buck and make me the most secure.

Guerra: So one of the things you’ve written that you focus on is education of stakeholders regarding the cyber risks of connected medical devices. Do you have any general approach you take or advice you can give to others on how you handle that education?

Mustac: So I like to, within the group that I’m working with, look at the popular devices that they have. Look at the known risks and vulnerabilities and what’s been happening with them, and then talk through what are we doing to make sure X cannot happen in our environment? Or if it did happen, how would we react? You know, how could you deal with losing this device for an amount of time?

I think a great example is patient monitors. One of the things that patient monitors and many medical devices give us is efficiency to be able to support more patients with less staff. Your nurse ratio has changed depending on obviously the illness level of the patient, and then with the technology that you have. So you can have a nurse’s station where you have one person sitting there watching 20 patients at a high level that are moderately ill. And if you take that away, that one nurse can no longer do that job.

So how do you deal with it? Do you have replacements? Can you get them up and running quickly? Do you have trained staff that you can get up on that floor quickly to take care of people?

So you need to think through all those different scenarios. Where am I going to get the staff, and it’s not just throwing bodies at it, they have to be certified, they have to have credentials to get into the system. They need to know their way around to know where different things are. You can’t just take any live body and throw them in the room and say go to work. It’s not a simple job.

Guerra: So you’re trying to get them to think through the scenarios.

Mustac: Correct. That helps them. They’re going to think of their workflows and their downtime procedures. How do they need to address it because now I’m showing you that this is real, these things have happened to other organizations. So they’re going to think about when this happens, if it happens, how would I deal with it. And it also helps us on the IT side to get resources dedicated to step up our technology, to step up our processes and improve. But if there is no discussion, there’s no dialogue. It’s like the old, “If a tree falls in the forest and no one hears it, did it make a sound? “No, I didn’t hear it. Everything’s fine. I don’t hear anything. Right?

Guerra: Do they have to come up with specific BCP plans based on those discussions and deliver them to either you or their operational leaders? How concrete does it get?

Mustac: Well, it’s a collaboration, it’s to raise that awareness, to make sure everybody understands what can happen and what those probabilities are. And then we each go away within our workflows to improve them. And it’s an iterative process, we keep coming back to the table and keep practicing and walking through them to say, “Okay, my SLAs (service level agreements) are this. I can restore this function in this manner, based on these assumptions, what’s the probability of me having all those things line up when, if I need rental equipment or whatever, what’s the probability of that being in place?”

If it’s fairly good, then you may not need as much of a crutch on the other side, but you still need your Plan B, Plan C, Plan D. You can’t stop at Plan B. Because when things start unfolding, they can snowball on you, especially in a metropolitan region like New York City. There’s so much to think about. For example, if we were to go into diversion in New York City – even though it has a very small footprint size-wise – if you try to get from Harlem down to lower Manhattan during the middle of rush hour, even though it’s a three or four mile stretch, you might be quicker running.

Guerra:  One hundred percent.

Mustac: You know, I mentioned earlier before we started recording, that I had a commute from Queens to New Jersey for a long time. When I started that commute at the beginning of my career, it was a 35, 40 minute drive. And toward the end of my career as that area got more populated, it was taking me almost three hours to get home, especially if there was a Yankee game going on. It just became unbearable.

Guerra: Yes, I commuted in and out of New York City for 10 years from New Jersey – through 9/11 and the blackout – all those kinds of things, so I certainly know the challenges that can come up.

Mustac: The big joke with my family was when they asked me, “When are you going to get home?” And, I’d say, “Anywhere from 45 minutes to infinity.”

Guerra: Let me put an open-ended question in front of you to see where you want to go with it? What are some of the trends you’re watching? They could be threats you want to make sure your organization can handle or technologies you think you may want to leverage – that you say I think my colleagues might want to have this stuff top of mind.

Mustac: I would say two of the most concerning ones are quantum technology – being able to break passwords very, very quickly. I mean, that’s coming down the pike on us all very quickly, as well as AI and where AI can lead us. We’re seeing a lot of hype in the news about chatGTP and some of the other services, and how they give you these very well-structured responses. And people are questioning, you know, if I ask it to do something unethical, will it do it currently?

And in the experiments I’ve run with it, if you ask it to do something that’s unethical, it tells you it has this moral engine built into it, and it won’t do it. But everything with technology seems to be a matter of time. When people say something can’t be broken into, it can’t be done, that’s dropping the gauntlet, and people start going out there and trying to do it. Eventually, somebody will find a crack in the armor somewhere.

Guerra: So you want to stay on top of that stuff, and you’re testing it. And I’m assuming that would be something you would recommend to your colleagues that you need to get in there and play, so to speak, and see what this stuff can do. It’ll give you a better idea of what may be coming at you.

Mustac: Yes. I mean, that’s something that I’ve always done throughout my career as new technologies came out. I was always not so much an early adopter in terms of putting it into production, but an early adopter in terms of playing with it, tinkering with it. Let’s see what it can do. Is it marketing hype, or does this stuff really have potential in that it’s going to hit the marketplace one day and change things.

It’s just amazing the stuff that AI can do. And we’re seeing virtual reality in these headsets that people are using for training and different things. So there’s a lot of good potential, but every technology that comes out is a double-edged sword. As much good as it can do in the right hands; in the wrong hands – not to sound cliche from one of these superhero movies – it has the power to do just as much evil.

Guerra: That’s some cutting edge stuff. I’ve heard it said that it’s so much easier to write a phishing email. For the bad guys, that’s the low hanging fruit.

Mustac: Well, definitely social engineering and leveraging the human’s ability to make an error is always going to be a very, very frequently exploited endpoint, right? No matter how well-educated we are, well-versed we are, in technologies, we get tired, we get distracted, we’re trying to do seven things at once. And it’s easy to make that mistake. And I always pick up on this during the holidays, or when there’s a natural disaster of some kind, that there are groups that start to social engineer things immediately using that as a premise.

So it could be the earthquakes happening in Turkey or the war in Ukraine. And they know that people have strong emotions tied to this or, during the holidays, when we’re all waiting for those packages that we ordered too late. And you’ll get a text message that just says, “Your package is undeliverable,” right? You freak out a little bit, because you’re like, “Oh, no, my significant other’s package isn’t going to come in time,” and you click on that link. And then you might get distracted and go away while your machine is now compromised. So any one of us can make that error.

So that the biggest thing that I’m concerned about with social engineering is that there is so much information that is available about our organizations, about our leadership and us as individuals, people can learn a lot about you online if you’re not careful. They structure things to take advantage either of yourself, or someone that you’re working with, that works closely with you. They’ll call you up and they’ll say, “You know, Anthony told me to give you a call. And you know, can you cut this check or approve this invoice? Because he’s tied up and this vendor’s really angry. So we’d really appreciate it.” And depending on how they stage it, they may say like, “Yeah, I know, it’s important.” You know, if it’s 99% accurate, they may not question that last 1%. And they’ll click “go,” and then it’s all over.

Guerra: So you just have to be super vigilant. And I guess that’s part of the training is to slow down before you do certain things.

Mustac: I always tell people, don’t be afraid to challenge authority to verify something – even if they’re screaming at you. I’ve had people screaming in my face, beet red looking like they want to have a fist fight. But it’s like, “just give me two minutes, take a breath, I’ll go take care of it for you.” And then on the back end, you do your verification and 99% of the time, it’s going to be legitimate, but you do have to do those checks.

Guerra: And sometimes that pressure, that urgency, is a red flag. Right? Sometimes that’s the tell, like this person wouldn’t be you know, this intense about this.

Mustac: Yes. But I mean, if you’re in an organization that’s tight on resources and overworked, that might be the norm that there is a lot of pressure in the environment, right? So if they happen to pick an environment that’s a high stress environment, and they come in with that approach, people react. Well, they’ll be more accepting of that pressure saying, “Well, yeah, this is how we work. These things come down hot and heavy, you know, real fast.”

Guerra: Let’s talk a little bit more about the device vendors. One of the other things in your profile was you collaborate with the device manufacturers. You actually worked for GE Healthcare, so you were on the device side. So that gives you incredible perspective that a lot of IT professionals are not going to have. I recall a discussion with a CIO who said she was having a lot of trouble, I believe, with the device vendor. She wanted to make a change. I don’t remember the details, but she wanted to do something to the device from cybersecurity perspective because she felt there was a patient safety risk. The device vendors have reasons for where they’re coming from – there are certain things they’re not allowing you to do, although they may feel constricting. But I’d like your overall thoughts on the dynamics between the IT professionals at the health systems and the medical device manufacturers. What is your advice for having a positive working relationship with your vendors in that area? And can it be frustrating, sometimes, for security professionals?

Mustac: It can absolutely be very frustrating. The outlook for traditional IT folks is they want the speed, they want the expediency. When they see that a vulnerability is announced, if it happens to be a Windows device, there may be a patch already available. And if it’s traditional IT, you’re going to pump that out within a day or two and patch your whole enterprise.

If it’s a medical device running that same version of Windows, you can’t go take that patch and apply it yourself. And having that discussion, telling people you can’t do it is tough, because they’re looking at it from the traditional standpoint of, “Well, the machine’s in front of you, the patch is in your hands. Why can’t you apply it? I just did it to 30,000 servers, you know.”

But the complexity on the medical device side is the liability and who owns that liability? And the manufacturer is saying, “no,” because they’re on the hook. And they’ve got the device certified with the FDA and other entities saying it functions in a certain way. They tested it under certain conditions. And they put their stamp of approval on the test. “Absolutely. This is how it works.”

Now you come in as an outsider, even though it’s your device, and you put something on it, well you’ve tampered with it. It’s no different than if you buy a car and you go out and put a custom component on it, or you put a custom stereo in and you go back to the manufacturer and have a short, they’re going to say, “Well, you monkeyed with it, you did something, it’s your fault. Now you only go deal with it.”

And I mean, that’s a simplified example, we’re just talking about a simple car. But now, in the medical environment, the clinical environment, if you did something to it, and something happens to a patient, it may not have anything to do with the device, but we’re a very litigious society, and people are going to sue everybody. And they’re going to say, “Well, if you didn’t mess with that device, there wouldn’t be a question about the device now.” So we really have to be very careful with how we do it.

But the manufacturers are starting to come around. And I really noticed in the past year that that engagement is much better than it ever was. And they’re understanding that they do need to patch these devices. I mean, they’re getting a lot of heat from all the healthcare organizations across the country. And what I’ve always said to my friends on the manufacturer side is if you don’t do this, somebody else will, it’s a competitive world. So you’re going to see foreign competitors come in and start eating your lunch, because they’re going to come in with patchable operating systems to update security features.

So you can’t focus only on the clinical functions of the device anymore, you have to look at it from a more holistic standpoint of the environment it’s going to be working in and what the expectations are from the HCOs as customers to say, “Now, I do have to secure it quickly. I do have to react quickly when vulnerabilities come out.”

And I mean, ultimately, the solution that we have right now for a lot of these things that are unpatchable and unserviceable for the manufacturers is to segregate them on the network, to lock them down to the ports, protocols and IPs that they need to talk to. But that’s still not 100%, it still has some vulnerabilities in that you don’t have traditional XDR software or endpoint protection on a lot of these devices that other PCs do where if there’s an anomaly, the software will pick it up and it’ll stop it. We’re just limiting it to saying you now have three doors to get into this device from, but there’s still doors and they’re not monitored.

Guerra: Right. So it’s better than nothing, but it’s not the ideal. It’s certainly not, “Hey, no worries, we can just fix it this way.” Right?

Mustac: I mean, ultimately, you can also take some devices offline, depending on what utility you’re getting out of it. But that makes things difficult, again, because a lot of this activity came in from remote service ability and remote monitoring where they can help clinicians through an issue on a device without coming on site. So that takes minutes to deliver as a solution. As opposed to I’ve got to dispatch it to a technician when he’s free, he’s going to get in his car. He’s going to drive through Midtown traffic for an hour and a half, and then he’ll show it to you. So there’s a lot of efficiency that gets lost if you start taking connectivity away.

Guerra: Well, as you said, the device manufacturers may be getting better. But they’re still going to say, “Hey, we’ve got this FDA issue.” There have to be changes on that side as well. Does that make sense?

Mustac: Yes. And one of the things that I try to introduce in dialogues all the time between the manufacturers and the government entities I work with is, we need to draw a line in the sand to say, “Hey, going forward, we’re going to do these things to make things more secure, and by collaborating we will have standards moving forward. But there’s a ton of legacy behind us that is not going to go away for a very long time. So let’s just put down our guns and say, this is a problem we all need to fix together. And, you know, let’s look at how we’re approaching liability and who’s responsible for what and know that we’re coming to the table with the best intentions to deal with the mess that we have.

I mean, there’s a big legacy mess out there. And I don’t blame anyone for it. It’s just the way that the world evolved. We didn’t have these threats five years ago. So we need to look at it and say, “Okay, everybody bring your brains to the table, and what’s the best way to approach this? What’s the least intrusive way for to get the devices secured to the best of our ability, and let’s do it collaboratively.”

Guerra: And we’re much more concerned about the devices as entry points to the network, rather than being the end goals themselves, correct?

Mustac: Yes. So I mean, they’re using them as entry points. And a lot of times, I believe that they don’t even know where they’re getting to, they just find an open doorway. And they’re crawling through, they don’t know what’s on the other side of it, whether there’s a patient there, whether there’s a medical device, they just know they’re in the network, and then they start scrolling around.

But as far as targeting somebody with a specific device, I mean, there is a high level of expertise that’s needed to do that, and a high level of knowledge about the device and the person at the other end of it. If they were to attack a head of state, they would need to know what devices they have. Going back to the pacemaker example. To exploit a pacemaker, you need to be within a reasonably close proximity because they’re Bluetooth-based, you need to know the make, model, and serial number. There’s a lot of easier ways to address the problem (of getting in the network) for a lot less money and a lot less knowledge that don’t take a rocket scientist to do.

So that’s why I say, with everything, people look for the path of least resistance; it works with water, it works with electricity, it works with people. If there’s an easier way, they’ll find it. No one wants to go through all that trouble to try to hurt someone, they’re going to find a cheaper way to do it. But it makes a better Netflix show. It’s an amazing Netflix show.

Guerra: Have you been a consultant on any of these programs (laughing)?

Mustac: No, not yet. (laughing)

Guerra: They’re going to call you. That’s great. Tom, let’s just wind up with a final piece of advice for someone in your position at a comparable-sized health system, doing the same work you’re doing and you’re saying, “Hey, from all my experience of all these years, this is a little piece of advice to take away.”

Mustac: I’d say, just always maintain your focus; know what your endgame is, know what’s in your organization. Hopefully people by this point are getting very good at discovering what devices are sitting on their network, who owns them; and be inclusive, get the clinicians involved, get your building services staff involved, get your biomed folks involved, your IT folks. And some things that may seem like a monumental challenge to one group may become a much smaller problem when you involve others because of the diverse experiences and knowledge you bring to the table.

So definitely, just keep circling the wagons; keep focused on it. And it’s not all about spending millions and millions of dollars. Unfortunately, this stuff is very expensive, but not everything requires a silver bullet to address, so slowly chip away at it. Your security posture will get better, but you need to stay focused. Keep measuring, keep chipping and marching ahead.

Guerra: Great stuff, Tom. Thanks so much for your time today. I think people are really going to enjoy this.

Mustac: Thank you. It was great being here.