Q&A with Providence CISO Adam Zoller: Start with Sound Policies, Then Customize with Required Exceptions

Adam Zoller, chief information security officer at Providence, a California-based health system with 53 hospitals and more than 1,000 clinics, says as much as he’d like to standardize things like employee access and devices management in an organization that large, it’s very hard – because so many exceptions are warranted. Instead, he focuses on identity-centric security. In this interview with Anthony Guerra, healthsystemCIO founder and editor-in-chief, Zoller says that when it comes to service accounts, password vaulting and password rotation are absolutely critical to a zero-trust approach. Identity-centric security adds up to making sure that the people who enter your information ecosystem only access things that they should. The old mindset was to have a hard perimeter and a soft interior. “That doesn’t really work these days, because caregivers are traveling all around the world with portable devices,” Zoller says. “They’re accessing information systems with their phone. So again, identity is the new perimeter.”

LISTEN HERE USING THE PLAYER BELOW OR SUBSCRIBE THROUGH YOUR FAVORITE PODCASTING SERVICE. 

Podcast: Play in new window | Download (Duration: 36:12 — 24.9MB)

Subscribe: Apple Podcasts | Spotify | Android | Pandora | iHeartRadio | Podchaser | Podcast Index | Email | TuneIn | RSS

TOC 

• Creating a culture of security – key partners
• Taking a zero-trust approach
• The challenge of managing identity (for human and non-human accounts)
• The medical devices conundrum
• The benefits of reducing ecosystem complexity
• Third-party risk management

Guerra: Adam, thanks for joining me.

Zoller: Thanks, Anthony. Appreciate being here.

Guerra: All right, looking forward to a fun chat. Can you tell me a little bit about your organization and your role?

Zoller: Certainly. So, Providence is a hospital system out on the West Coast of the U.S. We operate in several states. In the West Coast, we have 53 hospitals and 1,000 plus clinics, about 125,000 plus full time caregivers, and then another 100,000, plus non-employee caregivers, providing care to our patients. My cybersecurity team rolls up to our chief information officer who rolls up to the CEO.

The CIO is BJ Moore. And then under my team, I have roughly 220 people performing security functions globally. So we’re mostly a geographically distributed organization with some centers of gravity in the Seattle area, Portland and Southern California. And then we have our Global Center in Hyderabad, India that does some shift work for our security operations center, as well as some functions in governance, risk, compliance, identity and access management, and some supporting functions in our office of the CISO.

Guerra: Very good. All right, I want to start with an open-ended question and see what’s on your mind. So what are some of the trends you’re looking at, or strategies you’re working on?

Zoller:  Absolutely. So I guess a few things come to mind. Number one is our culture of security, not just within the cybersecurity organization, but broader than the cybersecurity organization looking at the entire Providence org – instilling security practices into our business practices, or business processes, instilling security mindset into our caregivers, because our caregivers truly are on the front lines of the cybersecurity battlefield. They’re the ones that are receiving phishing emails, they’re the ones that are making decisions on what they click on, what they don’t click on, interactions with our clinical device vendors, or clinical application vendors. They’re making risk choices every day. So informing them about security, training them on security, and instilling security culture – broader than just the security organization – has been a real focus of ours this year.

Another focus of ours has been on implementing or continuing the journey, I should say, toward a zero trust approach here at Providence. And when I say zero trust, a lot of people use the term, “never trust, always verify.” And basically it’s identity-centric security, making sure that the people who enter our information ecosystem access things that they should have access to, and don’t access things they shouldn’t have access to. And treating identity as the new perimeter versus the old mindset within security, which was to have a hard perimeter and a soft interior. That doesn’t really work these days since our caregivers are traveling all around the world with portable devices. They’re accessing information systems with their phone. So again, identity is the new perimeter. That’s a strong number two: zero trust approach.

And then number three is becoming a business enabler for the broader Providence organization. I want security to be a differentiating feature for why our patients seek care at Providence facilities. It doesn’t mean that security has to be front and center. But security and usability in the clinical workflow definitely needs to be front and center for our clinicians, so things like touchless authentication – we call it frictionless authentication. When a clinician accesses an information system, they shouldn’t have to sit there and take their PPE off and type in their username and password. That just doesn’t work when you’re trying to provide high quality patient care. So things like that.

Guerra: Wow. That’s a lot. That’s great. So let’s start digging in: security culture. When you talk about that, one of the things that comes to mind is, who do you need to work with? I’m thinking marketing – could be other people – but in your mind, to instill a security culture, who are the key individuals that a CISO needs to work with?

Zoller: Certainly communications and marketing make sure that when we do market, whether it’s internal or external, that it contains security-centric language or language that doesn’t put us at odds with our security strategy. But really, it’s the security leaders, the people who make risk decisions and influence risk decisions at the executive level across Providence. So our chief financial officers organization or supply chain organization, human resources, it absolutely plays a role in that. The legal office plays a role in that, our chief risk officer plays a significant role in that, from a risk compliance, privacy standpoint.

And the way that we govern is we have a governance body, an information protection committee. And as much as everyone loves to hate committees, it is pretty effective to get those people together on a periodic basis, talk about the bodies of work that we’re driving forward from a security standpoint, both tactical and really strategic topic-level bodies of work, and gain consensus, build alliances through that organization to drive these things forward. Because I can’t, for example, implement a security control like multi-factor authentication across our ecosystem without letting HR know it’s happening, because I may run afoul of some HR regulations for people who have disabilities to access systems, just as an example. So I need consensus, I need people to be aware of the security changes that are coming in. I found that by creating these governing bodies, like this council, it’s really made it a lot easier for me to drive security forward and Providence.

Guerra: You talked about working with HR, you talked about identity. I read recently there’s lots of technologies out there that will help you manage identity with the onboarding and off boarding of employees. And I read someone talking recently about the importance of security, working with HR to know when employees are being let go. That’s got to be a very tight relationship so those credentials can be revoked and that access is no longer there. What are your thoughts around that?

Zoller: Absolutely. HR is a really tight partner of ours – because, as you mentioned, when people join an organization, leave an organization, they move roles, every single one of those HR actions has an identity element that’s tied to it. We want to make sure they have their identity ready for them on Day One, so they’re productive on Day One, in a secure fashion. And what that means is that they have things like Office 365 available for them on Day One – that they know their username and password, and they know the service desk phone number if they have issues with their username and password. That they’ve set up multifactor authentication before Day One, so they know how to log in on Day One and have access to the systems that they need access to.

When they leave the organization, we want to make sure that we revoke access to our information systems in a timely fashion. Because if somebody leaves the organization and joins a competitor, for example, we don’t want them to have access to our data. And nor do we want them to have access to really any data. Even if they didn’t join a competitor, we don’t want them to be able to log into our systems after they leave the organization for a number of reasons.

And then think about the use cases when people move roles, which happens all the time. If I have a person who’s in information technology that has what we call privileged access or heightened level of security access to our applications and systems, and they move to a role in finance where they don’t need that heightened level of access, I don’t want them potentially clicking on phishing emails with a heightened level of access, or their security tool access or something like that. I don’t want them performing their day-to-day job with that heightened level of access.

Guerra: What’s challenging about doing identity right? I mean, it’s easy to say, “Oh, this is what we need to do. This make sense. Identity’s the new perimeter. And if we shore this up, we’re going to be in pretty good shape.” But I’ve talked to other CISOs, and it’s hard when you get into the granular level. Tell me why it’s hard and maybe some ways to go about it that you think will work.

Zoller: Identity and access management in the health system is very hard. And it’s very hard, especially when you’re in a health system that operates in some of the states that we operate within. We have to work around the foundation structure in California, where our physicians can’t be employed, they have to be contractors, independent contractors, as per state law. It makes it really challenging to manage what they should have access to, what they shouldn’t have access to when they join, when they move. If they only access our systems once a quarter, how do you, for example, set appropriate timeouts on accounts and account deletions if a physician only comes in once a year, once every six months?

And then we have people who may teach at our university part time. So they have a university account and access with the University of Providence, but they are a credentialed physician, and they practice medicine in one of our facilities. So they’re a physician and they have access to Epic. And then they also hold some other role. And managing those accesses, managing those roles, becomes very difficult. And as you can imagine, in a large organization, you can have some level of standardization on job families between regions, for example, different levels within my organization, or different pillars within my organization, they have different accesses to different things. But there’s a lot of personal tailoring for individual identities that goes on, because I may need access to or I may have the same access, same role as somebody sitting next to me, but may need separate access, different access to different systems.

So we try to automate as much as we can through some automation systems on the back end. We’re a Microsoft Azure shop, so tying our Azure Active Directory to our other systems that people need access to, we try to automate as much as possible and standardize as much as possible. But realistically, there’s going to be some human element to discern what accesses different people need access to on day one. And so I have a fairly substantial identity and access management operations function where I have humans that field tickets, and they crank through those tickets, and they make sure that people get access to what they need access to. And we get a lot of tickets, by the way. It’s about 2000 tickets a day. So it’s not a small number.

Guerra: So there are many exceptions, right? We build these families of permissions based on job descriptions so at least we have a starting point, but then the exception tickets start flowing in. That’s just the way is has to go, right?

Zoller:  It really is, yes, and you can use data to make some of these decisions and be as targeted or tailored as you can be. But realistically, you’re going to have to have some sort of a manual function to comb through the exceptions and grant access to things that people need access to.

Guerra: We know it’s the case, but it hit me when you said that every single person with credentials on your network is a point of possible compromise. That’s a lot of risk for any large organization.

Zoller: Potentially, yes. And they’re also a strong ally for us in the security organization to point out issues. We get a lot of user-reported issues, people who notice things that need to be fixed that they report to the security team. So yes, every identity, and therefore every person, has some level of exposure and potentially introduces some risk into our organization. But it’s a double-edged sword. They’re also very strong allies.

And I say, from a training and education and awareness perspective, let’s take full advantage of our caregiver population, give them the tools that they need to be able to report issues when they see issues, educate them on what’s right and what’s wrong. I’ve run across in my career, a lot of times (not necessarily at Providence, but previous organizations as well) where people do things that are against policy, but they didn’t know what the policy was. So I find that educating people on what right looks like is not only a strong deterrent for people who are setting out to do the wrong things, but it’s also a strong enabler for people who want to do the right thing.

Guerra: So identity’s the big thing, and we have identities of human beings. And we also have identities of devices, correct? And we call those endpoints, right? Human beings can make mistakes. There’s a question of the credentials, which we already discussed. They get their credentials, we want to make sure they don’t have more privileges than they’re supposed to, and that the privileges change as appropriate as they change through jobs and then revoke when they come off. So we want to manage that identity of the human beings. We have identities of devices, and that’s another huge issue. Right? Especially the medical devices. So is that a way that you think about things – is it overall identities and then we split it down from there into buckets?

Zoller: Well, identity-centric security, certainly. As a practice or as an approach, that is the core of zero trust. From a device or a machine account perspective or an automated perspective; we call them service accounts – accounts that don’t necessarily have a human tied to them but they perform important tasks throughout our environment, managing those is just as important as managing the human identities. In fact, in some cases more important because they may have an elevated level of privilege to do whatever task they’re doing.

Say it’s a privileged service account that runs a script on a periodic basis to pull sensitive data from one system out and do some calculations and then upload the results into another system or transfer the results into another system. That account’s always sitting there with that level of privilege, potentially readily accessible by a threat actor at any point in time. So the approach that we take to secure those accounts is password vaulting, password rotation. So you can use technologies that are available commercially to vault those credentials and make it so it’s only usable by the systems that should be using it for the specific tasks that it should be used for.

But it’s also incredibly difficult to change the passwords on service accounts. Because you can imagine, you can write software or write code around a specific service account to do a specific task. And then you may have to actually update the code to get the service account password to change. Or if you change the password on the service account, it breaks the code, and then all of a sudden the business process is broken. So all these things are interconnected. These systems – this particular technology we’re using internally – will allow us to vault the credential, but also rotate the password, change the password, on a periodic basis, without breaking the business process associated with it. There’s a number of technologies that do this out on the market without naming any specific technologies. There’s a number that do that. Password vaulting, password rotation, is absolutely critical to a zero trust approach.

Guerra: Now are medical devices included in that bucket of service accounts, or is that another bucket?

Zoller: No, certainly, medical devices use service accounts. There’s human accounts and non-human accounts, really two basic types. And clinical devices and clinical applications could use a combination of both service accounts to apply patches on the clinical device or transfer data between the device and an application. Or you could also have human accounts that are logged into the clinical device to do remote service or any number of things.

Guerra: There’s a tremendous amount of talk around the security on these medical devices, is this something that is top of mind and an issue that you’re involved in as well?

Zoller: It absolutely is: clinical device cybersecurity is one of the issue areas that – not a lot keeps me up at night – but this has the potential to really cause some scary outcomes if you think about it. The healthcare sector has a tremendous reliance on third parties to provide us with secure clinical devices and the ability to keep those devices secure and patched, updated on current operating systems. And oftentimes, these devices are fully managed by vendors or third parties for a number of reasons. And they’re certified by the FDA, and there’s a number of checks and balances they have to go through.

But what we find, or what I found, is that these devices, they’re designed to be deployed within hospital systems for up to 15 years. But if you think of a commercial operating system, like Windows, for example, it’s only designed to be within its lifecycle for about five to seven years, maybe 10 years if you stretch it and you pay extra for additional security patching. So there’s automatically going to be a gap there where you have devices designed to be in your system for 15. And then the OS is for seven to 10. So managing that is incredibly important for hospital systems to think about.

And what we’re doing at Providence is we use a couple of technologies to not only fingerprint the clinical devices throughout our ecosystem, but to detect vulnerabilities on those devices. And then we’re proactively going out and working with those third parties to make sure they’re patching those clinical devices in a timely fashion. But it’s challenging, it’s a really challenging problem to solve. You think about hundreds of clinical device manufacturers and vendors just at Providence alone. Thousands globally, I’m sure. And every single one of these devices has their own unique software bill of materials that it comes with. And to be completely frank, a lot of these clinical devices made by those manufacturers weren’t really designed with security in mind, 5, 10, 15 years ago.

So I think there’s a lot of work to be done in clinical device security. Some of it’s going to come as a result of legislation and regulation. And the rest is going to be, I think, driven by the customers, which is the healthcare systems, which are demanding, like Providence, secure clinical devices and the ability to secure those clinical devices with modern security measures.

Guerra: Are you going to try retire some of these devices maybe sooner than you would have, because they present a higher risk?

Zoller: Some of it will realistically. For example, if there’s a device that’s FDA-certified, and it can only run on Windows 7. Windows 7 is at end of life with Microsoft, and they’re not going to provide security patches forever on Windows 7. So realistically, if we can’t use that device, if it has security vulnerabilities, and we don’t feel comfortable running that device depending on what it’s been used for, and if we can’t virtually isolate it, we will retire it and we will replace it with a more secure device. Depending on, again, if the device touches a patient, the types of procedures that it performs.

We have a risk ranking system that we apply to the devices depending on what they’re used for, and we replace those devices, we lifecycle those devices, based on that. And we also prioritize patching on devices based on that risk ranking system.

But to your previous point on virtual isolation, we are taking that approach, as well. The technology that we use to fingerprint, detect vulnerabilities on the devices, we are also using that technology to propose virtual isolation measures, access control lists, and apply those access control lists to the devices using another technology that’s in our core backbone for our network. But to your point, it’s not easy to do that at scale, because every device is different, every facility is different. And as much as you’d like to have standardization across your facilities, doing it at 53 hospitals and 1,000 clinics is very, very difficult.

Guerra: And you can’t even find them half the time, right?

Zoller: Well, we can find them. And we have hundreds of thousands of clinical devices and we find all the devices. The challenge sometimes is working with the third parties and getting them to adopt modern security practices, to be completely honest.

Guerra: Has this changed the buying process of medical devices? It’s interesting because the regulatory mood is changing. There are things coming out from the FDA, they’re going to require a software bill of materials. So things are changing quickly. But I would imagine, just like third-party apps, there’s a whole new process for purchasing medical devices that I assume includes security on the front end with your team. So how has the buying process changed, knowing what we know now and what the industry is going through, How has it changed the buying process for medical devices?

Zoller: I think that’s the key; you hit it, which is you want to fix what’s already in your environment, but you can’t let anything insecure into your environment while you’re going back and fixing the sins of the past. And so, to your point, the buying process has changed dramatically, both in the language that we use in our contracts, which include our business associate agreement, terms around cybersecurity measures that our third parties need to conform to. And then also the specific contracts that we negotiate with our third party clinical device vendors, which include provisions like if you sell us a clinical device, it needs to be patched and current and updated through the lifecycle of that device (to the point earlier on operating systems being live for seven to 10 years, roughly, and then the device being live for 15 years). Well, I don’t want to buy that device if you’re not going to upgrade it through the lifecycle of that device. Because you can’t say that it’s good for 15 years if the operating system is only good for seven, that doesn’t make any sense. So anyway, that’s going in the contract, and that’s being negotiated upfront now, and talking to my peers, the CISOs across the industry, they’re doing the same thing. They’re working with their legal shop, their supply chain organization, to make sure that security provisions are baked in at the front to make sure that their expectations are understood before they buy a clinical device. And I’ll say the major clinical device vendors are taking this seriously and they’re willing to work with us. But it’s going to be a journey getting to a secure state, a completely secure state across the industry.

Guerra: It sure is. I think you’ll have plenty of demand for employment for the next however many years you want to work.

Zoller: I’ve been trying to work myself out of a job for 15-plus years.

Guerra: There are lots of bad guys out there trying to make sure you stay employed, unfortunately. You mentioned before about the possibility of changing a password on a service account and breaking a business process. Many are trying to get away from a complex environment by rationalizing or reducing their application mix. CISOs seem to be especially supportive of this. What do you think?

Zoller: At Providence., I’m blessed to work for a CIO and a CEO who are big believers in simplification, and keeping things modern, both from a security standpoint, but also from an end user, usability standpoint. And within that simplification play is standardization around security configuration, that standardization around process. It also makes our caregivers lives so much easier when they know what’s an approved device or an approved protocol already. And they can go out and procure a capability that they know isn’t going to have to go through a huge level of security scrutiny or another architectural review that’s going to take weeks or months or whatever. It’s a big enabler for our caregivers.

So I would agree that complexity can create some serious issues. Realistically, though, working in a hospital system, you’re going to have to deal with some level of complexity. So the question is as a CISO, how do you scale your capabilities, scale your processes to leverage the scale of the business and not try to necessarily centralize everything to the point where you’re acting as a roadblock to everybody? So relying on vendors, for example, to provide accurate architectural diagrams up front to us so that we can just review their architectural diagrams and not have to create them as a result of a new device being on-boarded, just as one example. But yes, I agree. I mean, complexity makes things hard. Simplification is the way to go both from a cost and usability standpoint, but also a security standpoint.

Guerra: The idea of perhaps having to go through security is fairly new. And people are trying to get their arms around it. And I know that for some people, it’s very difficult, the volume is tremendous. If you work for a large organization, you can have a tremendous volume of requests coming in from the business of apps they want to get or technologies they want to use, or they want to buy, and everyone needs it. You want to manage those in a timely fashion so that you’re not seen as a bottleneck. So how do you handle that?

Zoller: I can see why it would be a problem. The way that we manage it, though, is a little bit different. I think a lot of security organizations who – just like you said – they become the recipient of this giant firehose of requests coming in from the business. What we do at Providence is we’ve centralized all of our security functions, or IT functions including end-user technical support, including the people that are at the elbow with the clinicians in the facilities, including our biomedical engineering teams.

We’ve centralized all those within the chief information officer’s organization. So my peers within the organization own all those functions. So educating them and working with them, partnering with them on what an approved device configuration looks like, what our security policies are, and security processes are, and how to expedite those requests that come through, has been a game changer for us.

The other piece is they know that the clinicians want these capabilities. But instead of having a conversation with them with people in the facilities about, “Hey, I want this app” or “I want this device,” the conversation has evolved to, “I want a capability, here’s the capability I want.” And then, because we’ve centralized everything under the CIO’s organization, we can reach back or people that are at the elbow can reach back and say, “Do we already have an app that’s approved, gone through a security assessment already, that fits this capability?” And maybe it’s not the one that the clinician wanted necessarily, or saw, but it gives them the capability that they need. And they don’t have to go through any security review whatsoever. It’s just a matter of provisioning access.

But look, the volume of third-party security requests is very real. We do hundreds of those per year. And it’s cumbersome, it’s dozens of questions that have to be answered upfront, and then likely an architectural review. And getting accurate security architecture diagrams is a challenge, getting accurate answers to the questionnaire is a challenge. And then going back and fielding exceptions that need to be managed as a result of those third party risk assessments is a challenge. It all adds latency to the process that, frankly, when you’re providing patient care, you don’t have time for.

So I feel the pain of our clinicians and people that provide care. On the other hand, it’s just not acceptable to introduce security risk into our environment without at least understanding it and putting some mitigating factors in place. So yes, it’s a challenge.

Guerra: Last question because we’re about out of time. But one of the complaints that comes from the requester, in this case, is a lack of visibility into where does it sit in the evaluation? So I don’t know if you have any thoughts around that. But it seems like transparency would help if there was some dashboard where they can see, oh, it’s sitting here, or it’s stuck in legal because sometimes it’s not stuck in security.

Zoller: It is a real problem. Yes, I’ve heard I’ve heard this pain from people on the ground. Again, we’re blessed to have all of our people that are at the elbow and within our facilities that perform IT functions and related functions in the same organization. So from tracking within contracting, supply chain, legal, they’re all well versed on that, and they know who to talk to, to get a status update from legal, for example, or from supply chain.

Ideally, you could track this stuff centrally in one location, but I haven’t seen that done in a seamless fashion, really anywhere. But I would say from a security standpoint, the pain point that I hear oftentimes is that the security architecture teams don’t usually sit within the governance risk and compliance (GRC) organization, and the GRC performs third party risk assessments. So you have these two assessments that need to be done within cybersecurity, to pass the cybersecurity checkbox. And the frustration point that had been brought up to my team in the past was, “Well, we did a security review, and it passed. So why can’t we go alive?” and they didn’t realize, well, it passed third party risk, but it didn’t do a security architecture review.

So it’s exactly what you’re talking about, which is people don’t know the status. So what we’ve done within cybersecurity specifically is instituted, and my team’s going to hate me for this, I know there’s another name for it, but I still call it the One Intake Process. So you can submit a request to the security team. It comes through one centralized intake, and then they have tracking through some of our tools that we use for request tracking, they have tracking for the status of that particular request through the security third party risk review through the architecture review. And they know who’s fielding it, what the status is, when they can expect it back, all in real time. So that’s just within the security organization, though. That doesn’t solve the coordinating legal and supply chain and whatnot, but at least I would say the people on the ground within our organization know who to talk to to get that done.

Guerra: Okay, very good. Definitely another challenge, Adam, I’m going to give you a chance for a final thought. What’s your best piece of advice for someone at a comparable-sized health system in your position.

Zoller: Yes, look, my friends that are in these positions, it’s a tough position to be in. But I’ll say, ransomware is scary. These attacks that we’re facing are very scary. The threat is very real. But where I’ve seen people get the most bang for their buck is just doing the basics while investing in patching, investing in asset inventories, building strong advocates and allies within the business, understanding the business and speaking and communicating in terms of business outcomes, and then building allies and then gaining traction with people who accept and make risk decisions in the organization.

I mentioned that we have some councils that we set up but also gaining allies. Again, I’m blessed to have a board that’s very supportive, a CEO, CIO, that’s very supportive. Having the support of individuals in those places of power makes your life so much easier. So I’d say build alliances, make friends, and do the basics, and then you’ll be dealing with just the small sliver of remaining outliers from there.

Guerra: Adam, thanks so much for your time today. That was a great talk.

Zoller: Thanks, Anthony. Appreciate it.