Q&A with Northwell Health CISO Kathy Hughes: “BCP Requires More Preparation Than Ever.”

Having been a victim of identity theft, Kathy Hughes, VP/CISO for Northwell Health, is probably more sensitive than most to the possible effects of data breaches. As such, she and her team work extremely hard to make sure they don’t happen; but to be in a position to respond quickly and efficiently if they do. And being efficient with one’s reaction, to Hughes, means having processes queued up and ready to go when something happens, not using that as the starting point to put a response team together. In this interview, Hughes also talks about the importance of IT not being isolated in the organization, how important it is to prepare new hires for immediate phishing attacks, and how to address employees who can’t seem to keep from clicking.

LISTEN HERE USING THE PLAYER BELOW OR SUBSCRIBE THROUGH YOUR FAVORITE PODCASTING SERVICE. 

Podcast: Play in new window | Download (Duration: 33:22 — 61.7MB)

Subscribe: Apple Podcasts | Spotify | Android | Pandora | iHeartRadio | Podchaser | Podcast Index | Email | TuneIn | RSS

Bold Statements

“ … you could be down for days, weeks or even months and you really need to be prepared to deal with that.”

“ … ultimately what we do and how we protect our systems isn’t an IT issue, it’s an enterprise issue.”

“It just takes one person to click on a link in order for systems to become unavailable. Disciplinary action can lead up to and include termination.”

­

Guerra: Hi Kathy, thanks for joining me.

Hughes: Thanks for having me, Anthony.

Guerra: Great, looking forward to having a nice chat. I love to start these out, it’s an interesting role, you’re not just in healthcare, you’re not just in IT, you’re in healthcare IT security. So I like to find out how people wound up where they are. Take me back and tell me how you wound up in this little niche of the world.

Hughes: Okay, w­ell I think like most of my peers and other people I know that you’ve interviewed, we tend to start out on the infrastructure side of the house and the operations and the engineering, even the architecture side. Which is where I started many years ago. I’ve always had an interest in security. It was part of what we did on the infrastructure team. Typically, people associate things like patching and malware protection with security, but it became much bigger as time progressed. A lot of organizations, like Northwell, felt the need to break it out separately and to really focus time, effort and people on really building out a program to ensure the appropriate defenses were in place.

When I joined Northwell initially I was on the outsourcing side of the house and in charge of all the traditional infrastructure teams. Then an opportunity became available at Northshore LIJ, which is now Northwell, to really lead up and head the disaster recovery program. I really started focusing on that about eight or nine years ago. Built out that program to where it is today and we’re still going through a number of changes as things evolve and such. During that time, I was asked to temporarily take on the security function because the director in charge of the group had decided to leave. I took over the team on an interim basis and quickly learned just how much needed to be done in that area.

When they did eventually hire the director to take over, I specifically requested that I hold onto the risk management group because I really found that interesting and knew that a lot needed to be done specifically in that area. So I held onto that and really matured that program, as well. Really built our HIPAA compliance and security program (along with my peers in corporate compliance) built out the PCI security program and other programs, as well.

That was a real significant accomplishment. From there the role of the CISO had become more well-known. A lot of organizations were starting to add CISOs within their organizations. I ultimately applied for the position and, as they say, the rest is history.

But really what sparked my interest in the area was the fact that I was actually the victim of identity theft. Having lived through that firsthand and just realizing how vulnerable you feel when you’re violated in such a way where your identity has been compromised, it really made me want to focus that much more in this particular area. I feel very fortunate that I’ve been able to not only take my experience in the infrastructure role that I had for many years, being able to build upon that. But also taking my personal experience that I had having been the victim of identity theft and learning firsthand what people had to go through. I was able to leverage that and build out the program that we have in place today.

Guerra: Is identity theft as much of a nightmare as people say it is?

Hughes: It really is. The identity theft that I had experienced years ago before it was really a popular term, it was really learning from the ground up what it felt like and what the obstacles were and all the different things you had to go through with reporting to local police and law enforcement. Because I had been erroneously charged for things I didn’t buy, dealing with the credit card companies and the firm that claimed I had made these expenses and trying to get that removed from my credit rating was something that was really a very painful and very long process to go through.

Then just learning from there things that you had to put in place, such as making sure you have credit freezes in place so that somebody can’t compromise or can’t impersonate you to make purchases like that or to use your identity for other fraudulent type purposes. I’m just using this as an example, but healthcare services or filing tax returns or things like that, which is what identities are typically used for. They’re not only used for credit card fraud but for other purposes, as well.

It’s really important and that’s one message I always try to get out there, is to encourage people to put credit freezes using the big companies, Equifax, Transunion. And then if you do need a line of credit then to go through the process to have that freeze lifted temporarily while you get whatever transaction you need to get done completed.

Guerra: Sometimes you see these breaches and you see a couple million records were released. You wonder if people feel like, well, I’m one of a few million, nobody’s going to look at my records, nothing’s going to happen. But you’re more sensitive to that because it happened to you. I’m assuming you would be much more proactive in trying to make sure if there ever was a breach that the records that were exposed, that those people did everything possible to make sure nothing resulted from that.

Hughes: Absolutely, I feel very strongly about that, and I’m very concerned about that. That’s why, and I’ve shared this story with not only other people but my team as well, because it’s really important to have them understand the meaning behind what they’re doing and that it is important and how disruptive identity theft or any kind of misuse of someone’s personal identity, just how impacting it can be. Luckily, I was able to catch the identity theft pretty early on, so it wasn’t – I’ve heard horror stories about things that have happened to people where they’ve lost their life savings and things like that. People don’t know what they need to check and monitor to make sure that there aren’t any fraudulent charges or misuse of their user ID and passwords. I try to encourage people to practice those things.

It’s important, I think, to kind of provide that insight to people and let them know what it’s really like to have been a victim. I think that’s been very empowering to me and very important to the people that I’ve shared the story with.

Guerra: I hate to say I guess it’s an upside of what happened to you, right? At least you’re doing something constructive with it. I want to go back to disaster recovery. You mentioned that was part of your background before you even – when you were still doing infrastructure and things like that. I think a lot about how closely tied security has to be to disaster recovery because we’ve seen organizations have to go to paper and then have to go back.

As I think through what that’s like, it has to be so incredibly complex in terms of the interactions between IT security, IT and the operational leaders, the clinical folks. You are dealing with the outage, you are dealing with the breach and possibly a ransomware situation, whatever the case may be. You may have to take systems offline. You have to understand the operational impacts of what you’re saying may have to be done. I’ve also been thinking about how much CISOs have to know about the clinical operations so that they can be well informed when they’re talking about risk, when they’re talking about different things and, for example, this system may have to come down. Well, do you understand what that system does and the impact of that system coming down and have you coordinated with the physicians, have you game-planned that out?

Anyway, talk to me a little bit about what proper disaster recovery and business continuity planning requires in terms of IT working with actual clinical operational leaders.

Hughes: That’s a great question and it’s so incredibly important. I think traditional disaster recovery in my roots really came from planning for the very unlikely scenario that there would be a flood or fire at a data center and you’d have to recover systems offline. Typically organizations did that through making sure they had offsite backups or they could build up servers at a co-lo or a location that maybe they had on contract. Today we all know that it’s much more than that because the likelihood of a flood or fire, although still real especially in certain areas, is less likely than being attacked by a cyber incident, as you’ve alluded to. The need to recover systems quickly is very, very important.

At Northwell what we’ve actually done is we created a separate group from me, but I’m very closely aligned and even have a dotted line reporting structure to the senior vice president in charge of our business continuity and crisis management group. We have a separate division just focused on that to deal with all the things you’ve just mentioned. It’s really understanding what are your most essential services that you have and what would you do if they were impacted.

Most, I’ll call them business continuity or downtime procedures, are really focused on short time incidents, being down a couple hours because a piece of hardware broke or being down because there’s some type of an upgrade that’s being done. I’ve had to work very closely with the senior vice president of this group to really make them understand that you could be down for days, weeks or even months and you really need to be prepared to deal with that. This has been really something that is fairly new to us as far as the longer-term planning. We had very good, solid short-term plans in place but we’re starting to really build out that program a little bit more. We’ve engaged a third-party consulting firm to kind of assist and guide us with this.

It’s important to not only understand these essential services, as I mentioned, but all these systems and data are connected to everything else. So you really have to understand your upstream systems and data feeds into your primary essential services as well as what those downstream data integrations need to be and how that impacts your downstream applications. Everything in healthcare is connected. Having the full view of a patient’s clinical information is critical, and that involves all these systems and the information from them being available. I think this is an area of focus for Northwell as well as for other organizations.

And in parallel to that we are transforming the way we do traditional disaster recovery. So we’re looking at technology platforms like continuous data protection where we’re always replicating data to an offsite cloud area or to an off network type backup solution so that, should we be hit by a cyber incident, we have the ability to recover in the most timely and efficient manner possible. Those are just some of the initiatives that we’re currently working on and the tremendous progress we’ve made in those areas, but of course there’s more to do.

Guerra: It’s like an insurance type situation. You could spend all kinds of money so that you could recover within a millisecond, but that would cost a ridiculous amount of money, so what’s appropriate?  What’s the expectation for recovery?  When do we want to recover?  Then you might do your research and say here’s what it’s going to cost, what do you think?  And they say it costs too much. Okay, let’s say it takes a little longer to recover, can we bring the price down?  So where do you start?  Do you start with the expectations of the business for when it expects to be up?  Or do you start with here’s what it’ll cost to get us this level of service?  There’s different ways to go about it, right?

Hughes: Yeah, so it’s really a mix of the two. Because I think that nowadays we all expect, and this is no different in healthcare, everybody expects everything to be available all the time. But the reality is that that comes with a price tag. What we’ve done is we’ve established what we call disaster recovery tiers, where we’ve gone through a process to really look at all the applications in our environment to really understand which are those that are really essential and most mission critical to the organization and to make sure that we really invested appropriately in those so that, should we need to rely on that, if we need to bring those systems up within a short period of time because the expectation is high 90% availability, that we have the ability to do so.

So we’ve gone through this disaster recovery classification process. The typical spread that organizations have is that most of your tier one applications, that should be the far and few between, 5% of the environment. Most applications that you have, about 80% might fall into a tier two. And then a tier three might be things that are important but not so critical, maybe 10%. And then your remaining 5% would be if this system goes away and never comes back or we don’t get it up for weeks or months, we’ll be okay, we have other ways to support the operation and continue to work.

It’s really going through it carefully because everybody’s going to tell you – if you ask someone how important is your system they’re going to say mine is the most important system.

Guerra: That’s what I was just thinking. I was going to ask you how do those tiers get defined. Because if you’re just waiting for that list to be delivered to you everything would be in tier one.

Hughes: Yeah, so we came up with very clear definitions. Tier one is a mission critical system. These are the systems that must be available because if they’re not someone might die or we might lose a lot of money. I’m just kind of paraphrasing. Without them our organization may not be viable if we’re down for a significant amount of time. So it’s really, to your point, it’s engaging not only the IT folks but it’s engaging the clinical people because I certainly don’t know what systems they rely on most heavily. And things change over time, too, because a system that might have been deemed mission critical at one point isn’t so important now because a new system they brought in has comparable functionality built in.

There’s a constant review cycle that needs to take place to make sure that your applications are tiered appropriately and that you really have identified through a committee-type approach what those most essential services are so that you can ensure that you can make them as available as possible.

Guerra: I’ve read things here and there that say the IT department’s going to go away. Somebody wrote a thought piece about that, that it’s just going to be part of the organization. It makes me think that in order to be a successful healthcare IT executive today you cannot be in any kind of isolation. Because you need to be so involved with the operations and understand so much about what’s going on and coordinate and communicate. That old idea of IT being in a little bubble in a separate building, it just doesn’t work.

Hughes: I totally agree, and I would add to that collaboration is key. My main role, I would say, is making sure – although I report to the CIO, I have dotted line reports to corporate compliance and internal audit and corporate security, which handles physical security, and our corporate risk management team and our office of legal affairs. I work collaboratively with them on everything that we do. I’m involved in several committees, a couple of which I lead. We have an IT risk governance committee that involves representatives from all those different areas that I just mentioned and also our research organization, too.

It’s something that we continually talk about. It’s my job to make them aware of what’s going on in the world and what the current threats are and what we’re doing to protect ourselves. But, ultimately what we do and how we protect our systems isn’t an IT issue, it’s an enterprise issue. It’s a risk management process that we need to go through to make sure that we’re setting our priorities appropriately and that they align with the overall strategies and vision of our organization to make sure we’re investing appropriately.

It’s investing in not only process and technology, but people. I think that’s a really important variable. One of the concepts that I’m always trying to get across to everybody is that everybody in our organization is a member of the security team. It’s not just me and my few departments that I have. Everybody has a role and responsibility. One of the most effective ways that we’ve combatted phishing, as an example, phishing attacks, is that we have given and empowered our users with a button to report suspicious emails if they think something’s suspicious.

Because of their vigilance and because of the awareness that we’ve created, we have thwarted many attacks from being successful (even after) they got through our technology filters but, at the end of the day, it was up to the person behind the keyboard making a conscious decision or just looking at an email and saying, “Hey, this doesn’t look quite right, I’m going to report it.”  Many times it’s just spam emails, marketing emails that come out. But on occasion there have been some that have gotten through our filters and because people took the time to report the emails being suspicious we were able to put the appropriate controls in place to block access to a malicious site or to remove unopened emails from other people’s mailboxes that might have also received it.

Guerra: Right, so still the biggest threat is coming in through email and it’s social engineering and they can be more sophisticated. They can be pretty sophisticated. They’re doing research. They’re targeting particular individuals, they know who they are, they know where they work, they know the department, they’ve studied them on social media and they’re able to reference things that they’ve put on social media. Or they allude to things going on at the organization, new construction or whatnot. Sometimes they’re targeting people in accounts payable, where the bad guys are pretending to be some vendor or things like that. So these are the ways that things are getting in. You’re doing everything you can from a technology point of view as a strainer to get rid of a lot of it, and what comes through you hope employee education covers the rest, correct?

Hughes: Yes. We conduct, as many organizations do, phishing exercises to really test people’s understanding of how to detect and report suspicious emails. We do enterprise, we call them, campaigns or exercises, but we also do targeted exercises, which is really critical. So, for example, every few months we target our finance department who has wire transfer capabilities. Because, to your exact point, they are very highly targeted individuals, their names are commonly known through LinkedIn or through some posting that they might have on social media. A threat actor might be able to determine that they have certain capabilities. So they will specifically target those users.

So we will conduct phishing exercises. First we always provide some training and reminders and things like that, what to look out for. But then we’ll actually test them. We’ve gotten to a point where we work very closely with our HR department, our human resources department, and legal, and also with our compliance department to create a disciplinary matrix for people who, in these exercises, repeatedly fail or fall victim to them to reinforce how to recognize and report suspicious emails.

It’s a progressive matrix where we’ll provide training, on the spot training. If they click on a link they get a page that’s flashed up. If we see repeated behavior, we’ll make them watch a video specifically on phishing. If it happens a third time we might require them to do a real, live, online – it used to be classroom-type training but we don’t do that now, and do something to really reinforce the concepts and really work with the individuals.

That’s really the important thing. We want to work with these individuals to make them understand how important it is and what happens. It just takes one person to click on a link in order for systems to become unavailable. Disciplinary action can lead up to and include termination. That’s how seriously that we take this.

Guerra: I read that you had said that and I was impressed. I ask a lot of CISOs about this kind of thing and not many people like to talk about getting tough. Everybody wants to talk about in healthcare we’ve got to get along, tell them to do better. But at some point I say if somebody’s repeatedly doing this, first of all you might want to study them in and of themselves and try to figure out what is this profile of a person who keeps failing? Can you work this into the hiring process? Should this be part of an interview, something someone’s asked to get a sense of whether or not they have any concept of this and are sensitive?  Do they get it, or they blow it off and you go, we’re not sure we want this person in our organization. 

Hughes: Well, during the hiring process itself we really don’t have the opportunity to do that other than some kind of criminal background check on an individual. But what we do, though, is we have, as part of our onboarding process, I have my manager of our security awareness and training program give a talk. Actually every Monday morning there’s a new-hire training and she is on the agenda to speak to this group to explain the importance of phishing, how to recognize and report phishing emails. And she also tells them, “And by the way, hint-hint, within the next few weeks we’re going to test your knowledge of this.”

That’s really important because, to your point, new employees are the most susceptible. When we ran our enterprise campaigns and tried to figure out what type of targeted exercises we need to run, we found there was a very common theme between new hires. New hires just didn’t seem to get it because they hadn’t been barraged with our constant screen savers and email blasts and newsletters and digital signage and posters and social media posts and all those things we use, using mobile communications. We have partnered with our corporate communication groups and asked them what are the different ways that we can reach out to people. There’s different methods we apply to different individuals. Some people will only look at email and other people will never look at email, they only look at social media posts. So we’ve really leveraged all those different communication methods to try to get through to our staff.

So new hires that come on board haven’t necessarily been exposed to that. That’s one of the ways that we really reinforce that concept. If they fail the first time around we will re-phish them to make sure that they get it and work with them one-on-one as needed. It’s much easier to deal with the smaller group and give them that personalized attention than just throwing a phishing net out and trying to just get one message across to everyone. It’s more effective to target users that are most targeted because of their role in the organization or those that are most vulnerable because of their stature, specifically new hires.

Guerra: Do you try and keep them fair, those phishing emails?  You don’t put anything in there that you’ve got a bonus, because I’ve heard people get upset with that.

Hughes: No, no. But what we do, though, is we do have a recognition program. Just like we have a disciplinary program we also have a recognition program. In the course of a year we do four to six enterprise campaigns, if there are people that always report that the email was suspicious then we will recognize them with either a cyber champion badge of some type or even with what we call My Rewards points. We have a system where you can earn points by doing certain things.

I remember I always wondered is this really an effective strategy. I remember a couple years ago when we were able to walk around our office buildings. I was walking around and it was right after we had just notified people that, hey, great job, we’re giving you these reward points, they’re going to be available. I was walking by and people didn’t know who I was. They were so happy. A couple people had just received this email, they were so happy and thrilled and patting themselves on the back about, I got these reward points and isn’t this great. Word of mouth just got out about it and they were being our advocates and telling their coworkers, you really should report these things, you can get these points and you get this nice badge and you get this nice email. So it really has been a very effective strategy to not only recognize good behavior but also to call out those individuals through disciplinary actions that need to have behaviors reinforced.

Guerra: Anything come to mind if I ask you what is something you’re looking at, either a technology, a service, an issue going on that you say maybe not all my CISO colleagues are quite up on this or maybe I’ve done research and I’ve got a way to move forward and I think there’s a lot of uncertainty around this?

Hughes: Yeah, I think there’s a lot of buzzwords out there. I think that there was even a lot with this recent Log4j vulnerability that was discovered back in December. All kinds of marketing information from different vendors of, we’ve got the silver bullet to solve your problems. We all know that there’s really nothing out there. What I suggest is really making sure that you’re able to press that button when you need to get all hands on deck to address an issue when it’s as large as this Log4j vulnerability that was discovered.

I would just advise them to make sure that you have the ability to do that, to make sure that you can press that button to assemble people together, to explain to them what’s going on and what you need from them.

We have well over 1,000 applications within the environment. This Log4j vulnerability, you just didn’t know when it first was announced whether or not your applications were even susceptible to this vulnerability. So we really needed to engage all our application teams and say, we have a call to action, we need you to reach out to your vendors and find out if the application susceptible. And let’s take a risk-based approach to this, let’s focus on those systems that provide the most essential services to our company, to our organization, and make sure that they’re aware of this, number one. That they’re doing something about it, number two, if they’re susceptible. Meaning are they working on a patch and when do they expect to have it available.

We did that and we still have work to do but we’ve tackled those systems that are most critical ones. We did involve people from all our different application teams and asked them to kind of step up and work with us, collaborate with us to get this vulnerability mitigated. So that’s one thing that I think is really sometimes overlooked. I think that, to your point, sometimes security teams or traditional IT teams tend to work in this bubble and they can’t. You need to get everybody involved and engaged when things like this are announced.

Guerra: You mentioned an all hands on deck button. Are you talking about something more specific or concrete?  Or are you talking about relationship building to where they know who you are, you’ve spoken to them before?

Hughes: We have a service management team. Part of our service management team is a situation management group. If there’s a hurricane coming, if there’s some kind of major upgrade being done on a system, if there’s some type of major outage because a system isn’t working for whatever reason, our situation management team which we have already in place gets involved in providing communications, getting information from people and then communicating out information. So we leverage that team, our situation management team.

I reached out to the head of that department, I said, “We have a situation and we need to get ourselves organized around this and we need to get representation from all the different application teams.” We also needed our infrastructure teams because infrastructure teams own applications. As well as our key contacts in departments outside of IT that actually manage systems. For example, a clinical engineering team. We needed to get all these folks on a call, explain to them what’s going on and we needed to have some kind of weekly cadence to review status and to report out to senior leadership what’s going on and what’s being done.

That’s what we did, we just leveraged that team that we already have setup. It was really not meant to deal with this specifically, but it’s the same concept as if there’s a hurricane coming and we need to make plans we get this group together. So we just leveraged that same group for this purpose.

Guerra: So your advice is to make sure you have a situation management team, that’s your all hands on deck button that calls them?

Hughes: Yes.

Guerra: I like it, very good. All right, Kathy, that’s about all we have time for today. Fantastic conversation, lots of good stuff in there. I really appreciate your time.

Hughes: Well, thank you for having me. It was my pleasure.