Published November 2022
These days, some health system CISOs wonder where the buck stops when a cybersecurity event goes down. Who’s responsible for helping the clinical side go through the motions when the CISO says it’s time to go to paper? It should be the CEO, COO, CFO and/or other top executives, and the entire process needs to be planned and practiced until your organization is bored, says Erik Decker, vice president and CISO at Intermountain Healthcare. In this interview with healthsystemCIO Founder and Editor-in-Chief Anthony Guerra, Decker says the enterprise-wide response to a cyberattack comes down to muscle memory. “Because then when it’s muscle memory, if and when the event happens, you’re going to be able to get over all the original decision-making burden and be able to manage the context of the specific incidents that you have in front of you.”
LISTEN HERE USING THE PLAYER BELOW OR SUBSCRIBE THROUGH YOUR FAVORITE PODCASTING SERVICE.
… ideally, if you have a new contract coming in that’s for technology or data services, that should go through a supply chain; a supply chain should have a fork that comes directly to you. And that way, you don’t have to go out there chasing everybody all the time.
The majority of threat actors out there are criminal organizations that are looking to make money. And they’re doing that through extortion and extortion via ransomware, etc., they’re going to go for the lowest drag pathway possible in order to do that. And that’s why basics is the thing that you have to make sure you’re up to snuff on.
… SBOM is good. It’s the first step in understanding and unpacking what’s going on underneath it. But if you have to do that manually for hundreds of thousands of devices, you can’t. It’s just not enough. You don’t have enough people to do that.
Guerra: Erik, thanks for joining me.
Decker: Happy to be here.
Guerra: All right, great. Erik, do you want to just briefly give us a quick overview of your organization and your role?
Decker: Yes, you bet. So, I’m Erik Decker, I’m the VP and chief information security officer for Intermountain Healthcare. Intermountain Healthcare is an integrated delivery network located in the Mountain West region. So based in Salt Lake City, Utah, but we have Nevada, Utah, Idaho, Colorado, Kansas, and Montana in our portfolio. And being an integrated delivery network, we’re also both a health plan as well as healthcare delivery organization.
Guerra: Very good, Erik. All right, I want to just see what’s on your mind. So what are some of the top trends that you think CISOs should be working on or preparing their organizations to handle?
Decker: Yes, so the top issues that are happening today, of course, still start with ransomware. And there have been several big ones that have hit in the last 30, 60 days or so. And, you know, if you look at how these intruders are getting into these systems, it’s still the basics. Some of the trends that we need to be focusing on are our hygiene, our minimum standards, minimum controls that should be in place that work against the most prevalent attack vectors. And so it’s still multifactor authentication on your VPN, or the lack thereof on VPN, or email, remote desktop open to the outside world, critical vulnerabilities open on your perimeter. You know, these are things that are not overly complex, but the environments that we manage are really complex. And sometimes these technologies can work their way into the environment, you don’t even know that they’re there. And so having that good inventory, especially if you’re doing merger and acquisition work, you really need to make sure that at least those things are accounted for.
The other item that I still think about, and what we do on the national level, as part of the Cyber Working Group of the Health Sector Coordinating Council, is looking at third party and third-party attacks. So it’d be attacks, either against data that’s held by our third parties, or it’s third parties being compromised and a conduit of attack flows through that third party. So maybe you have just a critical supplier that has a VPN connection into your organization, a backdoor IPsec tunnel, something along those lines. And if those third parties are compromised, there are still plenty of intrusions that are happening via that vector. So, third-party risk and looking at third-party risk, not just from a sense of what data do they hold, but what access does that third party have to your organization.
Guerra: I assume every CISOs should be doing postmortems as much as they can when they read about a breach in the news to understand what happened, right? Then add to that, concerns about the basics, which as you said, are simple, but complicated, and it’s difficult to keep your arms around it all. What are your thoughts around that?
Decker: Yes, so I mean, it sounds easy. And unfortunately, it’s not, especially when you think about what a cybersecurity department does for an organization, you know, we’re trying to keep our lens on everything that’s happening, everything that’s changing on any given moment. Even if you have just a regular system that’s not going through growth, that system is changing constantly. And so cyber teams are out there trying to make sure that as this change has happened, you’re not introducing a vulnerability, you’re not introducing a new vector of attack, and so forth. And so you get spread very thin very quickly as that’s occurring.
You know, I mentioned third party. There’s a ton of contracting that organizations like ours do and so a lot of resources get dedicated to that. A lot of resources get dedicated to your 24/7 monitoring. A lot of resources get dedicated to your IT projects, etc., etc., etc. So, it’s easy to lose your lens on hygiene because you’re thinking about all this other stuff. However, most cyber attackers are getting in through VPN access, it’s compromised credentials, it’s the phish, although the phish is moving away, and it’s going to go back more towards vulnerability and exploit. But it’s your perimeter and what zero days or critical vulnerabilities are resident in your perimeter, and so forth. When you’re a smaller organization, you have limited resources. I would definitely say make sure that is locked up, and it takes resources to actually maintain that, too. So it’s not like you can just do it once and then you walk away, and all is fine. Some of that can change, you know, a request can come in and the change could be asked, and maybe it didn’t get cleared through security, and then you reintroduced a new hole. And so you’ve got to maintain vigilance on it.
For those groups that are doing merger and acquisition work, make sure you have a playbook in place, and make sure you have the top 10 things, the absolutes, that should be on that list. And so, as you’re bringing in organizations, acquisitions, and so forth, run those top 10 things. Look where two-factor is; it should be everywhere that is connected to the internet, and especially the VPNs. And make sure that you’ve got that in place. As you said, the postmortems, when events happen, we don’t always know how an organization was compromised. Sometimes that’s released through secure channels. Sometimes it’s not, it’s all dependent on the organization and their willingness to share that. So you can infer, based on how the event has happened and knowing what the common factors are.
Guerra: So just to be clear, you said even in a health system that is not going through M&A, there’s tons of change and morphing. You add on M&A, and that exponentially increases the amount of change, correct?
Decker: That’s correct.
Guerra: So when you were talking, I wrote down “process”, you use the word playbook. Right? I mean, these are the things that are going to help you; that you need to have process around. When this happens, these things have to happen. Right? I mean, that’s how we get our arms around these things?
Decker: Yes, and partnership with your supply chain organizations, your purchasing departments, your legal departments, your privacy departments. Those are all ways that you can keep tabs. So having official process. So ideally, if you have a new contract coming in that’s for technology or data services, that should go through a supply chain; a supply chain should have a fork that comes directly to you. And that way, you don’t have to go out there chasing everybody all the time. It’s just embedded in the system. And then you have your checks and balances. So you work with your partnerships, with your privacy teams, your legal teams. You educate them on the things that you should be looking for, and sometimes they will trip stuff for you that might not have gone through the official channel for various reasons.
Guerra: So there’s the basic attacks and the basic defenses. And then I’m guessing there’s the exotics. Do you have any thoughts around the cutting edge and what CISOs need to do? There are some crazy things that are coming down the road.
Decker: So you’ve got to think about what the intentions of the threat actors are before you’re looking to defend against their attacks. The majority of threat actors out there are criminal organizations that are looking to make money. And they’re doing that through extortion and extortion via ransomware, etc., they’re going to go for the lowest drag pathway possible in order to do that. And that’s why basics is the thing that you have to make sure you’re up to snuff on. If you don’t have that in place, it’s not hard for them to circumvent and overcome that.
And the idea that you’re too small to be known by them; that’s just not true. I mean, they’ve scanned the entire internet, they’ve scanned everything, they know who you are, who’s in your organization, they just know all of that stuff. So don’t rely on obscurity as a defense.
On the exotic side, we’ve seen elements of this happen. They have to expend resources to do it, and costly resources that hopefully get for a payout that they’re looking for. So you have to think about, what would that payout be for an exotic type of attack? You saw it with SolarWinds, that was definitely exotic. Where they embedded malware into the build of legitimate software that was deployed all throughout SolarWinds customers. That happened December of 20. I believe it’s two years now. And you know, why did they do that? Well, they were targeting government systems, as Russian speaking countries were going for it. And that was very likely espionage driven.
You’ve got to understand, why would they come at you with an exotic attack. What do you have that would be of interest in order to expend those resources? You keep the lens in play here. As we get better on the healthcare side, on our basics, and we close down all those doors, and we make it more costly for the attacker to come at us, they will pivot. And they will pivot. And they will get more sophisticated; they will expend those resources to continue on with their extortions and so forth. And so then it’s going to be more zero day-driven types of types of attacks. It could be more complicated social engineering attacks.
We’re already seeing multifactor fatigue attacks coming in, where they get your credentials, they try to make a connection, they can’t get in, because the multifactor goes off, and it goes off on your phone. And when it’s on your phone, they get denied, but then they just keep doing it until eventually you get tired of it, and you go fine and hit yes. So that’s going to come – and we’re already seeing that happened today. And so phishing-resistant or multifactor-resistant types of attacks are things that we’re going to have to start considering in the next five years, because I see that happening more often in the future.
Unfortunately, when you start getting into that realm, it’s less user friendly. So it’s things like UB keys. It’s going back to the hardware token that we had initially when multifactor came out in the early 2000s, late ’90s. With YubiKeys, you plug it into your computer, and that’s your cell chip. It’s user friendly in the sense that you can engage and work with the computer and it just happens, but you have to remember to carry it with you. And if you don’t have it, then what? And so you run into all these kinds of scenarios that make it a little challenging for the regular user to work.
Guerra: User friendly, but sometimes not so much if you don’t have it.
Decker: Yes. And that’s the rub, right? Is like how you make sure that it’s connecting in, and we do that via these proxies, multi factors and so forth. I would say if you’re using SMS as your second factor with codes being sent through a text message, that’s also going to go the way of the dodo in fairly short order. There are attacks that can circumvent that. It costs that attacker to spend a little time but there’s things called SIM swap attacks and such where you can clone the SIM card and then have those messages relayed to a different phone. And so, it bypasses the whole process when you do it that way.
Guerra: You know, you talk about “everyone’s a target.” One of the things that I’ve heard, and I believe makes you a bigger target of a specific focused attack, is probably research. If they think you’re doing something with nation-state level interest in research around maybe COVID vaccines or whatever, that can really put a target on your back, so to speak. Does that make sense?
Decker: Absolutely. You know, especially if your organization’s doing DoD level research or they have federal contracts in place, that would be a target of interest by not necessarily criminal organizations, in this case, now you’re talking about nation states and espionage and IP theft. So you look at different countries that are interested in doing that as they build up their economies.
Guerra: Right. Right. So you almost have to operate as a CISO at a different level when you’re running that type of stuff?
Decker: Yes, it’s a different risk profile, so you need to have an understanding of what that type of information is. What are the contracts that you have made? If there are federal contracts, then certainly there’s very likely something you have with the federal government. And so what are the terms and restrictions around that? If you’re an organization that is like pharma, or something along those lines, where the drug that you produce is your revenue stream. And so you get allowances with that drug before they become generics, and you spent a lot of time in R&D, if you lose that to somebody else, they can just produce that elsewhere. That’s a huge hit on your organization. So that’s a very different risk profile. It’s not locking up your systems through ransomware. It’s literally competitive edge. And so not every organization is going to have that, at that level of risk, but there are definitely some in this space in healthcare for sure.
Guerra: Right. Third-party risk, obviously, very interesting topic. You said that not only do you have to make sure that an entity has good cybersecurity, but you also have to check the VPNs that are coming into your organization, and what access others have. That’s fairly complex. I’ve heard people use the term, not only do I have third parties, but they have third parties, and they become my fourth parties. And then I have fifth parties. And it makes me think of that SBOM (software bill of materials) concept people are talking about. You know, with log4j, people didn’t know if they had it or not. So what do you think of that SBOM concept – it’s almost like a list of ingredients on software. Would that be something helpful to security professionals?
Decker: Yes, so SBOM, in particular, is the first step in a process for being able to understand what’s inside these products that we get from vendors where the intellectual property essentially is the product itself. And so you don’t get to look under the hood. So when the log4js of the world happen, we go scanning our systems, and we go looking for the vulnerability. And depending on the scanner that you have, it may or may not be able to detect it. If it can’t get into the system itself; it can’t authenticate to the system and look through the libraries and the software that’s installed. And so it makes guesses on stuff.
And so the SBOM intention is to bypass that, or rather give us the information upfront so that we can just look for it in one place. The challenge, the next step in the evolution of SBOM, is operationalizing it within healthcare delivery organizations. We have tens of thousands or hundreds of thousands of products that are in our environment, and to be able to maintain the versioning of software in every single one of those at any given time is going to be very challenging. So think about it, you have an infusion pump that has version X, Y and Z, you go out and do an update to that infusion pump of X, Y and Z and maybe half the fleet has been updated and half the fleet hasn’t. They have different versions now of that software. And so you have to know every version of every pump of every place and now compound that by every other modality that exists inside of a healthcare organization, and you’ve got yourselves a big problem, a big data problem.
So there’s there are folks that are working on solutions for this, and I’m hopeful that in the next five years or so we’ll see something come down that actually can do it at scale. Because this is absolutely a scale problem. Like I said, SBOM is good. It’s the first step in understanding and unpacking what’s going on underneath it. But if you have to do that manually for hundreds of thousands of devices, you can’t. It’s just not enough. You don’t have enough people to do that.
Guerra: Right. Let’s talk a little bit about business continuity. This is an area that I’ve been really focusing on trying to understand what CISOs should be doing. Is there a missing role in terms of somebody not overseeing this process at a high level? Let’s say there’s a ransomware attack. And the CISO, in consultation with the CIO, determines that we need to shut down certain applications within a very short period of time and go to paper. CISOs can take it to a certain point, but it’s not their role to make sure that the clinical side of the house knows what they’re doing at that point. So I’ll leave it for you there. That seems to me to be an issue.
Decker: Yes. So this should be in the basics – in the business continuity, in your incident response plan – especially the large scale incident response plan, and the drilling of that large scale incident response plan. So we start by offering some suggestions and solutions on how to do it and products that actually exist. So one of the other things that I do is I’m the chair of the Cyber Working Group of the Healthcare Sector Coordinating Council, which is one of the 16 critical infrastructures that’s been defined under the National Defense Authorization Act and National Infrastructure Protection Plan. It’s a construct that says industry and government have to come together when there’s critical infrastructure in place, because industry actually owns and operates the infrastructure. So within that group, we have just released the OCCI (Operational Continuity – Cyber Incident). And it is a checklist that was built in partnership with emergency management folks, cybersecurity, and emergency management folks on how you connect your cyber incident, a cyber disruption in with your standard emergency operations.
So every hospital has to have an emergency operations organization and plan. We have to account for physical disasters, fire, water, whatever. And so what you want to do as a CISO is connect into that process. You don’t want to create something separate. And so your access into downtime procedures, and all those other things, is through your emergency management department. So start there. And there are these things called incident command or hospital incident command, depending on which standard you’re using. There’s structures of command that are already built as part of that emergency operations, emergency management process. And you want to leverage those processes. So you’ll have an incident commander, you’ll have logistics, finance, marketing, media, public affairs, legal, and so forth. You build a plan, a cyber plan that accounts for the types of damages and impacts that can occur due to these kinds of ransomware attacks. Work with your emergency operators to figure out how you connect that into your incident command structure.
Build in the what-if scenarios you need to be thinking about. At what point are you going to take down your systems? When are you going to proactively take down your data centers and so forth? What would be the trigger points that would enact you to do that? And it’s very uncomfortable to obviously be postulating that, but you need to be postulating that before you have the attack and not making that decision on the fly. And then you drill it. So you go through this process you need to involve your executive leadership in this – certainly your CFO, if not other executive leaders, and you drill through the thought process.
Take just one of the things that’s happened in the news and apply that to your organization. Don’t even worry about getting super complicated about this. “Oh, I’ve got to have every little attack vector understood in order to do a proper tabletop.” You don’t need that. So what you need is to be thinking about, what is the event that’s occurring? Assume that it’s going to occur to us. Get people over the hump of, “Oh, well, that wouldn’t occur because of this thing,” and then arguing the scenario. You don’t want to argue the scenario, you want to work the plan, the scenario should be helping you determine if the plan is accurate or not. And then say, “Okay, how would we do this?” You know, is it one person that gets to make that call within a period of time? Are there two keys that have to be turned in order to do it? What do you do if that person is on a plane somewhere or unable to make a decision? What’s their delegate process? How do you go through all of that, and then you just update the plan, and you want to keep doing those drills over and over and over again, ideally, until people are bored with it. They’re like, “Oh, yes, we’re going to do this. We’re going to do that.” So it’s muscle memory. Because then when it’s muscle memory, if and when the event happens, you’re going to be able to get over all the original decision-making burden and be able to manage the context of the specific incidents that you have in front of you. So you’ll be able to leverage everything that you’ve worked on before.
Guerra: Excellent, Erik. I think unfortunately that’s all the time we have for today. But great stuff, great advice, and really interesting stuff about that disaster recovery. So I appreciate it, and I will talk to you again soon.
Decker: Thank you very much.