Q&A with CHOC Children’s CISO Joshua Roth: Tech Dependence Makes BCP More Important Than Ever

When it comes to getting a healthcare organization’s cybersecurity house in order, Joshua Roth, CISO at Children’s Hospital of Orange County (CHOC), says he starts with four things: the people, the processes, the technology stack, and managed services – and then looks to tackle the low-hanging fruit. In this interview with Anthony Guerra, healthsystemCIO founder and editor-in-chief, Roth talks about his new role at CHOC and what keeps him up at night. “If there’s one thing to pick,” he says, “it’s obviously any large-scale event. We train for it continuously. We’re constantly evolving to be in front of it.” Add to this a good bi-directional relationship and trust built with the business side of the organization, keeping the board educated, staff happy and not taking vendor security lightly. “My biggest risk area is the third parties and ensuring that they’re not compromised,” he says.

LISTEN HERE USING THE PLAYER BELOW OR SUBSCRIBE THROUGH YOUR FAVORITE PODCASTING SERVICE. 

Podcast: Play in new window | Download (Duration: 35:16 — 24.2MB)

Subscribe: Apple Podcasts | Google Podcasts | Spotify | Android | Pandora | iHeartRadio | Stitcher | Podchaser | Podcast Index | Email | TuneIn | RSS

Bold Statements

“ … one of the challenges that healthcare organizations have is your clinical space is getting younger, and their reliance on technology – and in all fairness to them, it’s due to the fact that we have all the technology here – is huge. If we go to downtime procedures, they have less experience or almost no experience of going into that method of doing their jobs.”

“One of my worst nightmares would be going to them and saying, “Hey, we’ve got ransomware. And I have no clue.” I don’t want to be in a situation where we feel like we’ve just got to shut everything down. In a healthcare organization, that is scary.”

“ … your resources – especially in cybersecurity where it’s tough to find them – are very, very important assets to you; the most important asset to you. So really making sure that, as you’re fighting the battle with threats, you’re paying attention to your people.”

Guerra: Josh, thanks for joining me.

Roth: Thank you for having me.

Guerra: Great. Looking forward to a fun chat. Josh, can you start off by telling me a little bit about your organization and your role?

Roth: So my organization, CHOC, is Children’s Hospital of Orange County, one of the leading hospitals in Southern California for treating children. And my role, obviously, is to protect that organization as the chief information security officer. I think it’s been about 10 months I’ve been in this role at CHOC, so I’m still relatively new to the organization.

Guerra: Excellent, excellent. All right, you want to tell me a little bit about your career journey, how you wound up in healthcare IT security?

Roth: Sure, so I’ve been doing security for quite some time. It’s probably been 20 years. I actually started out in networking in my journey within IT, and back in 2005, I worked for a city. And IT security didn’t really exist back then. So I’m one of those people that came up in cybersecurity as it evolved as an actual entity in an organization. So for most of us in that time period, we got to be part of the journey of what cybersecurity ended up being. So I started out in a city doing networking, and then with that had to own firewalls. And that’s when things were getting really interesting and it just kicked off my journey. I was very interested in it. I ultimately ended up at Kaiser Permanente. I spent a great deal of time at Kaiser Permanente. I started in 2007, and I went until about 2011. And then I spent some time in big four consulting doing cybersecurity and privacy, and then ended up back at Kaiser Permanente where I spent nine years before joining CHOC.

Guerra: Very good. So you said you’ve been at CHOC about 10, 11 months? Can you give me any insight into what you’ve tried to do in that time period to acclimate yourself to understand the state of cyber at the health system, what needs to be done urgently and down the road. But just your best advice for how anyone can start as a CISO in a new health system and the things they want to accomplish within the first perhaps six months or the first year.

Roth: I think the most important thing, from my perspective, is to know the business. Obviously, I was in healthcare for quite a while at Kaiser, but being the big behemoth that it was, I wasn’t as close to the clinical space being that it was such a large enterprise. So coming here was definitely a change for me because I was very much now involved in the day-to-day operations of an actual hospital. When I worked for Kaiser, people were always asking, “What hospital did you work at?” Well, I worked at the health system, but the only time I went to one of the hospitals was as a patient.

So with CHOC, I’m all about business enablement. I think it’s very important that I get to know the business and build that bi-directional trust and dialogue and collaboration with the business. So that was first and foremost for me, making sure that I have an opportunity to talk to all my key colleagues in the organization.

Secondary in the first 90 days as a CISO is looking at your people, the processes, the technology stack. Being an organization like ours, you’re going to have managed services, as well. So, looking at the quality you’re getting out of those and really assessing the overall picture with those four foundational items. So that’s the approach I took. I built a two-year, out-of-the-gate strategy based on what I saw in those areas. And I think it was important to do that, of course. And with that, we did flush out quite a few things.

First and foremost, I will say CHOC – in the absence of my position for a time – they did very well. I came into the organization, and I was pleasantly surprised by some of the security capabilities that they had in place. But there was also some low-hanging fruit and things that we jumped on right away that I observed, and thought were very important for the organization. So we were able to execute pretty quickly. That’s one difference here compared to a larger organization; we’re more agile here than I’m used to at the larger organization that I was at previously.

Guerra: Yes, I would imagine every CISO who starts in a new place probably wants to get in and find that low-hanging fruit, that really important critical low-hanging fruit. That’s probably something you want to do right away, I would imagine.

Roth: Absolutely. There was a list of things. The team was relatively small. I managed to get additional headcount right out the gate and do some hiring. That helped a lot. There were some technology stacks that either were projects that needed to get fast-tracked or reprioritized. I made some immediate investments, quite frankly, some technology stack that we needed, as well. So definitely important to look at the risk when you look at cybersecurity.

Healthcare in 2022 was a record year for us when it came to cyber threats against hospital organizations. I think it more than doubled. Children’s hospitals were obviously part of that target. A lot of children’s hospitals made news due to ransomware attacks in 2022. You think about Boston with the FBI involved in thwarting that attack from a nation state. There are a couple of hospitals on the East Coast as well, children’s hospitals that were greatly impacted by ransomware.

So that was another thing that I did coming in right away is definitely looking at the incident response. I held a very large tabletop exercise that involves the whole hospital business as well as part of my first 10 months in the organization. We did some penetration testing right away, making sure we don’t have any exposure out there that puts us at a greater risk.

Guerra: So you’ve had a very busy 10 months.

Roth: It’s been very busy.

Guerra: So you want to get the house in order, right? You want to get everything straightened out, any issues you see. Lots of stuff there. When you talk about building relationships and getting to know the business, we’re talking about the rest of the c-suite. To what degree are we also talking about the clinical leaders?

Roth: Clinical leaders for sure. Regarding the c-suite, I met quite a few of them during the interview process and got to talk to them a couple of times since then, but my focus really was in that clinical space and some of the leads in that space because they’re the boots on the ground. That’s really where I took my efforts to meet people and talk through any challenges that they’re having and what concerns them. I talk a little bit about threat landscape, of course, educating them and always taking an opportunity to educate and learn from them too, quite frankly.

Guerra: Right, and any consistent theme you hear back from them about their issues? Is it usually about having to log in five times, or getting kicked out of the system? That clinician satisfaction type stuff, things that slow them down.

Roth: Yes, one of the key areas, when I talk about business enablement, is that overall user experience. And so you hit the nail on the head there with the one item that you stated. So that definitely is one of the spaces. We’re customer-service driven. I want to have a good experience when our associates are using our tools, and they use our tools every day. Identity is a big one. You know, how do we make things easier for our associates in the organization when it comes to their credentials and how they log into systems and keep them moving through? That’s definitely a hot item there, obviously. We’ve made tremendous headway in that space, just since then. So I’m happy to report we’re on that journey there to make things easier for folks.

Guerra: You know, you talked about coming in and doing some things. I would imagine when you’re interviewing places as a CISO, you’re interviewing organizations, you want to know that they have a certain approach to security to be comfortable enough to work there. You want to know it’s respected, and you’re going to be heard. And it’s not just, “You do your thing, but stay out of the way,” type of thing, right? So how much can you find out about a prospective organization? What do you want to find out to make you comfortable as a security leader working there? We know John Henderson, your CIO. I assume you report to him. Great guy. He’s spoken on webinars for us before. There are certain things you want to hear when you interview. Tell me about that process? And if you can find out anything beyond that about actually what they have in place – to find out the maturity of the security so you aren’t surprised.

Roth: Yes. So I look at it two ways when interviewing for an organization. First and foremost, I have a personal connection to CHOC. And quite frankly, after my time in the larger healthcare system, it was very near and dear to me to remain in the healthcare space, preferably the nonprofit healthcare space. And like I said, I have a special connection to CHOC personally. And so, for me, it’s an honor to be in this organization, just given what this organization does for the community and for the families that are served.

So I will state that when it comes to looking at the organization, obviously, I’m not coming into an organization that is fully mature, where I’m expecting to just throw my feet up on the table and let things run. You know, I welcome the challenge. I mean, that’s been part of my career since I’ve been in IT. That passion to work on those challenges. So obviously, I wasn’t looking for an organization that’s ultimately in its most mature form. But to answer your question further, during the interviewing process I asked about the makeup of the team, where it fits in the organization. And I asked for an honest opinion on what the culture of the organization is. That was a big one. And then one of the key areas there is you ask questions about the budget. We want to know where cybersecurity falls within the budget and that data give you a sense of the priority of it. And the board; ask about the board members; ask about cybersecurity questions and the board. Ask about past presentations. How often you’re reporting to the board. I think those are very important, because that gives you a sense of how important it is for the organization.

Regarding the threat landscape, with ransomware there has been a tremendous uptick in healthcare organizations. They very much know. And if I’m getting questions from people in the organization, it’s actually a good thing, right? I’m very open door, if they’re sending me random emails asking me questions about threats, I don’t get offended if they’re saying, “Hey, are we cool with this one?” Because that means they’re paying attention.

Guerra: That’s a really good point that questions are good, board interest is good. As opposed to going in a place that the board never asked about cyber, because then you’re probably not going to get a lot of money in the budget. There might be correlation there, right?

Roth: Yes. And you know, the c-suite level, everybody is great. I actually met one-on-one with our CEO. I came in there with an agenda, and we ended up sitting down and talking about just cybersecurity in general. She very much shows that she has a vested interest in what we’re doing. And it was just a very informal discussion about all things cybersecurity, and even went into the how do I protect myself personally, and that’s obviously very important for some of the key individuals in your organization as you’re not just protecting them in their workplace. You’re educating them and helping facilitate reducing risk in their personal lives as well because it does cross over, especially at that level.

Guerra: I’m guessing that the conversations and the interactions with clinicians are completely different than they would have been five years ago. The level of understanding about cybersecurity from seeing these ransomware incidents where health systems are knocked offline for days or weeks. No clinician that cares about providing care and understands the degree to which it depends on the technology that they use every day can be indifferent to cybersecurity. It’s inconceivable. So I’m guessing you get a lot of interaction, a lot of support. Is that the case?

Roth: I feel I do when I go out into the business. Absolutely. You know, it’s top of mind to them when they’re speaking to me. So they’re asking the right questions. Of course, they’re always going to our phishing awareness training-type activities. It’s funny, when I talked to them, it’s like, “Oh, you tried to trick me.” And I say, “Well, I’m not trying to trick you, we’re trying to educate you.” We’ve even made a lot of changes there, you know, click rates in our phishing awareness training have gone down just in the past 10 months. I think the last one we did right before the holidays, we reduced the click rates on our awareness campaign emails by probably more than 50%.

Guerra: Wow.

Roth: Which is good. You know, when you look at the healthcare space too, and you think about the ransomware, I think what people will probably see is the fact that the organizations have to go into downtime procedures, and oftentimes to paper. And I think one of the challenges that healthcare organizations have is your clinical space is getting younger, and their reliance on technology – and in all fairness to them, it’s due to the fact that we have all the technology here – is huge. If we go to downtime procedures, they have less experience or almost no experience of going into that method of doing their jobs. So it makes things more difficult, in my opinion.

So I work very closely with our emergency response teams and our business continuity teams, and the top discussion is around some of the exercises in downtime procedures. Because we’ve got to make sure that this younger generation of folks coming in and doing clinical work here are prepared in the event that we have to go to paper in a downtime procedure situation.

Guerra: Tell me about the relationship between IT security, IT, and emergency management, whatever you call that department that’s supposed to make sure that the health system can withstand anything: tornado, hurricane, whatever. So do we think of an IT incident as having to tuck under and be one of the things that they are in charge of. So the CISO has to come in and say, “Hey, this is how a cyber event could unfold.” But they’re going to manage it as part of their purview of managing everything. So my real question is, to what degree is the CISO responsible for ensuring their organization is ready to handle a cyber incident, including downtime procedures? Do you tuck under emergency management, or do you lead it?

Roth: It’s a collaboration. So in preparation, they’ve got the team that runs the emergency and the business continuity. In fact, I just had a meeting with them yesterday. I’m at the table for those and so that’s obviously very, very important to make sure you’re at the table for those because it is going to be one in the same with respect to how we potentially respond. If systems are down, it doesn’t matter if it’s systems are down due to a cyber incident or systems are down due to the fact that a truck plowed into the electrical grid or something like t­hat. They have a command center. We pretty much have the IT command center. So I wouldn’t say we tuck under them. It’s more of collaboration.

I think the important thing here is that when you’re doing your exercises, regardless of what that exercise is, that all parties are at the table. So when I did the big cybersecurity tabletop exercise a couple of months ago, they played a very important role in that exercise. So I wouldn’t say we tuck under, but we definitely will be part of that command center and have our separate command center for the folks who are in the IT space tactically working on the response, the mitigation of a threat. I think it is important to separate those things so the people who need to focus on that stuff aren’t distracted by the larger group. So that’s the way w­e look at it. I think the larger organizations are probably set up very similar.

Guerra: One of the things I’ve been talking to people about, concerned about, is it falling through the cracks and nobody really running it in terms of making sure the organization is ready to handle a cyber incident where you have to go to paper. It sounds like maybe it’s just very important for the CISO to have a discussion with emergency management and say, “Hey, I just want to make sure that you’re considering a cyber outage in your plans – that it’s one of your scenarios. I’m going to help you. I want to work with you to work through this. But this has got to be a piece of what you’re looking at. It can’t just be tornadoes and floods.” Is that something that needs to be done? Or is everybody doing it? Am I concerned about nothing?

Roth: I would hope everybody’s doing it. We’re absolutely doing it. If they’re not doing it, I think they absolutely need to be doing it. My organization within IT, if a cyber-attack were to take place and we had to go into downtime procedures, I can’t manage the downtime procedures, that’s when you’re activating business continuity response in that emergency command center. So it absolutely is coordination and continuous dialogue, the muscle memory of it. And so when we have our business continuity meetings and I’m having my meetings separately, cybersecurity events are part of that conversation every single time. So absolutely, I think both sides, we collaborate continuously on what it would look like in an event. So if other organizations aren’t doing that at that level, they need to be doing that.

Guerra: What could a ransomware scenario look like in terms of your involvement? I mean, would it be where it pops up on the screens that you’ve been locked out of your systems, and you have to pay us a Bitcoin or whatever it is. You either see it, you get alerted to it. You’re now somewhat in the driver’s seat of ascertaining what’s going on, working with the CIO to figure out what’s going on, possibly call the FBI, calling emergency management, making a recommendation about what needs to be done. Do we need to take everything down? What’s interesting, especially in the discussions I get into with biomedical engineering folks who talk about, “Whoa, whoa, whoa, you can’t just shut off our devices. Somebody could be getting treatment on these right now.” So it’s just wild to think when you play this stuff out the different scenarios, the different conversations, you’re making recommendations and not decisions. You’re making recommendations to the CIO, who is perhaps then taking that up the line. Ultimately, this may be a CEO-level decision. Probably is. Just take me through a little bit of that, what your thoughts are there.

Roth: Yes. So you look at some of the events that have occurred just in 2022. You know, not naming the hospital, but when I created my cybersecurity tabletop exercise this year, when you look at that tabletop exercise, the injections that we put into the scenarios that we played out through that exercise, it looked apocalyptic. It was like, you’re talking about phones down, you’re talking about internet down and where you rely on Office 365, and you suddenly don’t have that. We’re not even communicating via Teams anymore. You know, you’ve got the EMR system down, all these scenarios happen. And going through that exercise, I know they’re thinking that this is impossible, that this would not happen, that it would not be this grand of an incident. Well, guess what? We literally mimicked what actually happened at a hospital organization just the month prior. So it’s not like we had this pie-in-the-sky grand scenario that would never happen. We actually mimicked what happened at another hospital.

When it comes to the responses to IT, some of it comes down to tools. So the technology stack is going to be very important. Can you isolate systems? Can you easily detect the blast radius? How quickly are your resources going to be able to understand the source of the attack or the type of attack that’s occurring. A lot of factors come into play just with your people, your processes and your technology. So your incident response plans, your playbooks, constantly exercising those and validating those along with your technology is going to be very, very important.

If med devices are down because of ransomware, that’s one thing. But if you’re deciding what you need to further protect and you’ve got med devices on the network that are operating, I think turning those off is not an option. However, from a technology perspective, you can do things like isolate. You have network segmentation capabilities. At the macro level, you’ve got endpoint solutions that can lock down systems or basically prevent that lateral movement of a potential threat in the organization. But understanding the blast radius of what occurred is going to be very important in that first 10 minutes, hopefully even sooner.

My goal with the technologies is to determine how quickly we can assess the impact to the organization so when I do go to the emergency management and to my leaders that we have an understanding. One of my worst nightmares would be going to them and saying, “Hey, we’ve got ransomware. And I have no clue.” I don’t want to be in a situation where we feel like we’ve just got to shut everything down. In a healthcare organization, that is scary.I want to be able to say, “Hey, ransomware started here, it’s hitting these systems, we’re locking those down,” and continually assessing to make sure we don’t have any more lateral movement, so we can isolate a threat early. It’ll minimize the impact to the organization.

Guerra: This is the job, right? I mean, you can’t be caught unawares when this happens. You have to have prepared your organization, done your best, documented things, to put the organization in a position for business continuity. I mean, how do we keep delivering care? A large part of that responsibility rests on the CISO, right?

Roth: Yes. When I’m in meetings, and even prior to me being in the position, anytime I was in a meeting with the CISO, somebody always asked the question, what keeps you up at night? And absolutely, if there’s one thing to pick, it’s obviously any large-scale event. We train for it continuously. We’re constantly evolving to be in front of it. But you won’t know until it hits. Obviously, it’s going to have some very important post-event lessons learned, and understanding some of the gaps that you need to address as part of that as well. So that’s part of the job, but being in this position, you have to evolve continuously, there’s no down day for you. You’re always going to be trying to stay one step ahead. And it takes a lot of effort with your people, your technology stack, talking to the organization as a whole, because the organization is kind my last line of defense too, I very much preach, “If you see something, say something,” even in the IT world.

Guerra: What would be one or two of the trends that in addition to what we talked about – ransomware, things like that – that you’re looking at trying to make sure your organization is positioned to deal with?

Roth: Email is still one of the top threat vectors. Our stack is really good there. But constantly staying on top of the threats that are involved in that space is important. But I think some of the key areas are probably going to be that supply chain, that vendor risk area, and how we address that, especially in the healthcare space. At the end of the day, healthcare organizations, like many organizations, rely on a lot of third parties. And that, to me, is going to be my biggest risk area, the third parties, and ensuring they’re not compromised.

Guerra: Is it tricky when existing vendors find out they are now being held to higher standards? And I’ve heard about situations where the CISO isn’t given enough time to review the vendor against new standards because the contract renewal is up in a few days, or something like that. Have you seen that?

Roth: There are cases where that happens. Luckily, the organization’s pretty in front of it on how we monitor our contracts. Those will happen, if it’s an existing vendor and it’s a renewal, and we are assessing their security and the risks associated with doing business with them. Just because it’s due doesn’t mean they’re going to be off the hook. I mean, especially for that existing vendor – because we’re already using their services or their applications – it’s not like I can shut things down, they’re part of the business processes, but it doesn’t mean that they get a free pass. We’re still going to assess them; we’re still going to work with them. And we’re still going to have to make a determination on risk acceptance.

Guerra: All right. Any piece of advice you would offer to your colleagues?

Roth: I talked about coming in and looking at your processes, your technologies, your managed services, but I think one of the key areas for me that’s allowed me to be successful is my people, as well. At Kaiser, for that nine years before I left the organization to come to CHOC, I had I think 14 direct reports with 0% attrition; nobody left me in nine years. So I think investment in your people is very important. And that’s really building that bi-directional trust and inclusion as part of that. And mutual accountability is very important. So your resources – especially in cybersecurity where it’s tough to find them – are very, very important assets to you; the most important asset to you. So really making sure that, as you’re fighting the battle with threats, you’re paying attention to your people.

Guerra: Great point and great interview, Josh, I want to thank you so much for your time today.

Roth: Thank you for having me.