It’s perhaps the ultimate case of foreshadowing.
A decade and a half ago, while working as a bedside nurse, Robbie Freeman started to become increasingly frustrated with laborious processes that took away from interactions with patients. Fortunately, he channeled that frustration into something productive by creating a digital process for medication requests. In addition to improving efficiency and enabling more time with patients, it also set Freeman on a trajectory to eventually become VP of Digital Experience and CNIO at Mount Sinai Health System.
Recently, he spoke with Kate Gamble, Managing Editor at healthsystemCIO, about the “zigzagged” path that led him to his current role, and the exciting work his team is doing to leverage digital and AI tools to “reimagine the way we work,” whether it’s by improving in-basket management, more effectively retaining information during shift changes, or making it easier for patients to access care. During the interview, Freeman spoke about the importance of starting small and “working out kinks,” how his experiences in clinical and quality improvement helped shape him as a leader, and what he learned by going back in the trenches.
LISTEN HERE USING THE PLAYER BELOW OR SUBSCRIBE THROUGH YOUR FAVORITE PODCASTING SERVICE.
Podcast: Play in new window | Download (Duration: 24:15 — 11.4MB)
Subscribe: Apple Podcasts | Spotify | Android | Pandora | iHeartRadio | Podchaser | Podcast Index | Email | TuneIn | RSS
Bold Statements
There’s a lot of synergy between the different areas with the work we’re doing in digital, in AI, and in informatics. And so, it does come together nicely. I’m always looking for opportunities to pollinate across the teams and make sure they’re supporting the important enterprise initiatives.
Generative and conversational AI are going to help get patients to where they need to go much quicker and more efficiently, and then we can have our team members focus on the highest value work.
We want to make sure that we’re asking questions and checking for bias, because we know that if you build these AI tools on biased datasets, you’re going to get a product that carries forward and can amplify that bias.
We want to get it right on a small scale, work out any kinks, make sure we understand it, and go from there. And that’s the approach we’re taking with GenAI. We want to get it right in smaller pilots and learn from that.
I always encourage them to try to see the healthcare system from different perspectives. I think it’s so important. It makes you more well-rounded as a leader.
Q&A with Robbie Freeman, VP, Digital Experience & CNIO, Mount Sinai Health System
Gamble: Hi Robbie, thanks so much for taking the time to speak. Let’s start with your role as VP of Digital Experience and CNIO — it’s not something we’ve seen a lot. Can you talk a bit about that?
Freeman: Glad to. The two roles really come together nicely. I’m responsible for a few different teams and areas. On the digital side, we have our digital experience team where we’re bringing to life the different digital products and solutions that are used by patients, and also really thinking through what are the tools that we want our workforce to have when they come to work.
I’m also responsible for our clinical data science team. We build machine learning and AI products mainly focused around patient care. We’re trying to predict different risk scores and safety events that can happen in the hospital; for example, who may be likely to fall or become delirious or end up in the ICU or be malnourished.
And then as CNIO, I have a team of nurse informaticists; we’re the translators between our clinicians and the technology teams. We have nurses based at our hospitals and nurses that support our service lines. I work as a dyad with our CMIO to support the clinical operations across the health system.
There’s a lot of synergy between the different areas with the work we’re doing in digital, in AI, and in informatics. And so, it does come together nicely. I’m always looking for opportunities to pollinate across the teams and make sure they’re supporting the important enterprise initiatives.
Leveraging digital tools to add value to nursing work
Gamble: It does make sense to marry the two, especially considering the many different ways in which various digital technologies can apply to nursing. So, as you know, there’s been so much emphasis on artificial intelligence in healthcare, but I want to focus on the nursing side. What are you doing and where do you see the most potential?
Freeman: Sure, I think that digital and AI have a lot to offer in terms of adding value to the work our nursing team is doing. There are a few things we’re excited about. One of those things is we’re expanding our virtual nursing program. We know that when we can leverage virtual care team members to assist with things like documentation and patient education, that’s a really good use of digital and virtual care.
Leveraging AI to “save time and streamline”
We’re also looking at ways that AI can help save time and streamline. Some of the streamlining is more of a manual process where we’re going through all of the different documentation tasks that we’ve added over the years and removing things. We’re tracking closely how much time our nursing team is spending in the system, and we’ve been making some progress in terms of eliminating all of the documentation tasks we’ve added over time.
To take it a step further, we’re working with Epic and OpenAI on a project that’s coming down the pike soon. Through our partnership with Microsoft Azure, we’re going to use GenAI to help summarize information, support nursing care planning, and support things like change of shift, where it’s really important to make sure all the key information is handed off from one clinician to the next. We know that’s a time where information can get lost, and things can fall through the cracks. So that’s one way where Generative AI can add value and help ensure we are keeping our patients safe and having efficient communication.
Spending time on “the highest value areas”
More broadly on the provider side, we’re also starting to pilot some of the capabilities to support the in-basket. As we’ve been focused on digital and expanding the number of patients who use our app at Mount Sinai and moved to a larger scale, we’ve seen a corresponding increase in number of messages. And so, we’re looking at ways in which Generative AI can help support that. Not that we want to take the human clinician out of the loop, but we think we can help with efficiency and getting rid of or automating things that are more administrative in nature. That way, we can spend our time on the highest value areas. I think that over the next six months to a year, we’re really going to see a whole lot of Generative AI makings its way into healthcare.
Conversational AI to provide “immediate” answers
Another example is the opportunity with conversational AI, which we launched on our website through our app, as well as in our call centers. We’re using that platform for a lot of the questions patients have, especially the less complex questions for which they may want to take a scheduling action. They may have a frequently asked question, or they may want to do things like refill prescriptions. And so, we’re building automated flows to help support it; that way, we can get an immediate comment to answer to the patient, and not have to step it up to human support, except for those cases that require an extra level of complexity. We think Generative AI and conversational AI are going to help get patients to where they need to go much quicker and more efficiently, and then we can have our team members focus on the highest value work for the areas that they’re supporting.
Taking an “experience-led approach”
Gamble: What you’re talking about are really practical use cases, and that’s one of the concerns people have with AI — that is focuses on vague ideas or cool technology. This seems like an area where it can really have an impact.
Freeman: That’s right. We want to make sure we’re spending our time and energy on areas that are pain points for our patients. And so, our team takes what I call an experience-led approach. What I mean by that is we want to co-design these solutions with our patients. We want to capture the voice of our end-users: our patients, our clinical teams, and our frontline teams along the way.
“Nudges and notifications”
I’ll give you an example. One of the things we heard during our focus groups was that patients would have a great experience when they come in and meet with our providers. But once the provider leaves the room, they feel kind of lost and think, ‘what do I do next? Where do I go from here?’ And so, we launched a simple checklist that we call ‘My Next Steps’ where patients can see if they have a follow-up referral, if they have medications to refill, or if there’s education. We build the nudges and notifications to help patients stay on track with those next steps in a simple checklist.
Putting patients “in the driver’s seat”
Another concern we heard is that when it’s time to seek care, let’s say when patients start having symptoms, it can be a little confusing figuring out what to do. Where do I go? Do I go to urgent care? Do I call my PCP? Do I do a video visit or go to the ED? There are a lot of options. And so, we also launched ‘Check Symptoms and Get Care,’ which is available through our MyMountSinai app, and that allows patients to enter symptoms. We match it using AI with evidence-based guidelines, and they get a recommendation so they can be in the driver’s seat in terms of where to go for care and why, and they can click into their appointment or get in the queue for an on-demand visit. We want to help patients navigate through that decision making.
Those are just some examples of how we’re being experience-led; how our AI tools are supporting patients along their care journey. We’re also being really intentional about ensuring that these tools are accessible to the diverse communities we serve. We want to ensure we’re really focused on getting things translated into the preferred languages our patients speak. We’re partnering in the community to make sure these tools are accessible.
Ethical AI
Gamble: That’s really interesting. Ensuring ethical use of AI is so important, and it’s something a lot of organizations are challenged with. Can you talk a little bit more about what you guys are doing there?
Freeman: Sure. Our health system has an AI governance structure, and we have set our own guidelines around what we expect both from AI products that are developed internally by our in-house teams, as well as the vendors we partner with. We want to make sure that number one, we understand how it works so we can measure the performance. We have a process to measure how the models work — and that’s true whether they’re coming from our EMR vendor, from a third party, or internally.
We want to make sure that we’re asking questions and checking for bias, because we know that if you build these AI tools on biased datasets, you’re going to get a product that carries forward and can amplify that bias. We don’t want that to happen. And so, we have a process where we can check for bias. We’ve done some work around algorithm bias and seeing how it works across different racial groups and whether there is variation. That’s been a collaboration with some researchers within our school of medicine. We’re continuing to refine how to ensure these tools are safe, free from bias; or that we can address the bias and that’s all done before they ever get used for patient care.
Gamble: Really interesting. So, the governance piece, as you referred to, is so important.
Freeman: Absolutely. It’s the only way to ensure we really understand how these products work before they make their way to patients. It takes, over time, building our internal capacity. It’s one of the items that we discuss with our cloud partners as well; partners like Microsoft who offer tools around better understanding how AI products are working within an environment.
Streamlining documentation
Gamble: Along those lines, how are you working to incorporate social determinants into the care plan?
Freeman: We have a health system effort that’s focused on social determinants of health. On the nursing informatics side, we’ve worked to make sure that the questions are easy to assess with our patients, and that when we do have areas that require attention, we can make it very visible through icons that are front and center for our nursing team. A lot of the work we’ve done is to make sure that the social determinants that need to be addressed are both visible and actionable and that we’ve been streamlining documentation so that we’re not asking multiple things in different places.
For example, there are questions around transportation needs. We didn’t want to have three separate places to document. And so, we’ve been aligning the documentation so that we can ask questions once and it will populate the different views within the EMR where we need to see that information.
We’re very much pro-patient reported information, which we do through our MyMountSinai app and other means of collecting information. And the goal is over time, we won’t have to continue to ask the same questions.
Starting on a small scale
Gamble: Going back a little bit to the work you’re doing in Generative AI with Epic, can you talk about where that is right now?
Freeman: We expect that in the coming weeks and months, we’ll have a number of use cases up and running. We’re still very measured around doing this on a small scale first. We want to get it right on a pilot scale and make sure we’re measuring and understanding the performance. We know that hallucinations are a thing with large language models. And so, we really want to make sure we understand it at a manageable scale. Initially, these will be smaller pilots, and depending on what we see with those initial results, that will determine how quickly and how wide we go based on the evaluations.
Gamble: Is that something you’ve done before just as far as co-development or working closely with vendors on these types of things?
Freeman: Yes. There are a lot of parallels with quality improvement. Before I was with the technology department, I was working in quality improvement and did some work with the IHI. They like to start with one patient-one doctor, or one patient-one nurse; get it right on a very small scale. I think the same holds true for our technology projects — we want to get it right on a small scale, work out any kinks, make sure we understand it, and go from there. And that’s the approach we’re taking with GenAI. We want to get it right in smaller pilots and learn from that; make any adjustments and scale it from there.
Gamble: It seems like a pretty good blueprint, especially since going too big, too fast doesn’t traditionally hold up a lot of times.
Freeman: Oftentimes, it’s not the technology that’s the barrier. It’s more around change management and getting buy-in to ensure we have adoption. That’s often the thing — how quickly can we scale? One of the projects is around how quickly we can make sure we have engagement and buy-in and the necessary intensity of change management for the projects that we’re working on.
Hybrid agile operating model
Gamble: You mentioned focus groups before. I’m guessing that involves patients as well as people on the clinical side. What has the approach been there?
Freeman: We’ve set up our digital team around an experience lab; that’s where we do our research, focus groups, and data and market research to come up with product concepts. That includes talking to the end-users. The example I have around ‘Check Symptoms’ involved a lot of talking to patients and consumers in our market to hear about their pain points.
Once we have the vision of what we want to bring to life, our digital studio does the actual development and building of the products, and from there we’ll iterate and continue to improve. We have what we call a hybrid agile operating model. We’ve tried to take a lot of the principles from Agile and incorporate that into the way we work. That helps us move at the speed we need to so that we can move forward with these initiatives.
From bedside to C-suite
Gamble: So, I’d like to talk a bit about your background. You started on the frontlines working on the bedside, right?
Freeman: That’s right. I worked as a bedside nurse within Mount Sinai Health System — my first six years was caring for patients. I loved working at the bedside and having those interactions with patients, but I did have some frustrations. A lot of them were around our EMR and some of the technology being used at the time. And so, out of my own frustration, I ended up taking on projects and volunteering to solve these things.
One of the first projects I worked on was to build a digital process for medication requests about 14 years ago. At the time, we would fill out paper forms when we needed something from the pharmacy. It was like a carbon three triplicate form where you’d write what you need and walk down to a window to wait for medications. It sounds so trivial looking back, but basically, I created a web form where you can fill out an order and it would print in the pharmacy, and someone would bring it back to the unit.
We measured and saw how much time nurses and nursing assistants had been spending trying to get medications from the pharmacy. Meanwhile call bells were going off. And so, we were able to reinvest into time spent at the bedside and ended up helping to support some work through an IHI program called Transforming Care at the Bedside, which looked at how to reduce the non-value-added time that’s keeping us away from patients.
“A view of the business side”
And so, I fell into this this world of quality improvement and then healthcare technology. Eventually the chief nursing officer offered me a job and then had the chance from there to work for the president and COO at the Mount Sinai Hospital and support many of our service lines like respiratory care while working in hospital administration. We started a data science team to support our ED. I was able to get a view of the business side of healthcare administration and run large departments.
And then about three years ago, our chief digital and information officer reached out about joining the technology department in the role I have today. It was a great opportunity to work at scale across the eight hospitals and the health system and go from supporting work at one site to system-wide, which has been a wonderful learning and growing opportunity.
Gamble: You can impact a lot of people now. But it seems like your time as a bedside nurse did really provide a solid background — not just in patient care, which is so important, but also in quality and getting these different experiences.
Freeman: Exactly. It wasn’t quite a straight line. It’s been a little bit of a zigzag, but it’s been a collection of experiences. Along the way, as I got more interested in data science and AI, I went back to a program at the NYU Business School, and then I went back to school again. Right now, I’m finishing up a doctorate program. So, I’ve come to look for opportunities along the way for learning and growing.
Gamble: I can see how the work you’ve done got you more interested in technology and innovation.
Freeman: Absolutely, and I think nowadays through our capabilities with digital and with AI, there are so many opportunities to reimagine the way that we work. Using these tools as enablers, you can start from a blank slate. And oftentimes not only are you able to improve the patient experience, you can also work more efficiently, and you can position the organization to differentiate and work more efficiently.
Growing up in the trenches
Gamble: And it seems like your experience has also shaped how you think about innovation and what it takes to drive that.
Freeman: Yes. I think growing up through the trenches has been helpful. I actually started my career as a volunteer, then worked as a nursing assistant and as a front-liner, and I was able to see healthcare at different altitudes. At each level, your perspective expands. That has been invaluable, because you really do get to see so many different perspectives of the healthcare system. When I talk to people early in their careers who are interested in healthcare and leadership, I always encourage them to try to see the healthcare system from different perspectives. I think it’s so important. It makes you more well-rounded as a leader.
Back to the ED
Gamble: That segues into another thing I wanted to talk about. I saw on your LinkedIn profile that at some point last year, you worked a shift in the ED because there was a need. Can you talk about what that was like from your perspective?
Freeman: I’ve always prioritized getting out there and rounding and being present. This particular incident happened about two years ago during a Covid surge when we were really challenged from a staffing perspective, and people were stepping in to help. I asked my team members to help out, and then I figured that if I’m going to ask them to do it, I need to be accountable myself. And so, I volunteered to support one of our EDs that was short-staffed. It came back very quickly, and it was like a wonderful experience just to be able to support folks and get to work with patients again. I got a lot out of it.
Gamble: Sure. And even though you’re not that far removed from your time at the bedside, I would think it was a great reminder to really always be thinking about those who are on the frontlines and making things as easy for them as possible.
Freeman: Yes. Many of our best ideas come from the people doing the work. That’s been a principle that I’ve really subscribed to. Last year, one of the members of our WOCN (Wound Ostomy Care Nurses) team, which focuses on patients with skin integrity issues, reached out and said, ‘I have an idea. Can we build an AI tool to figure out the highest risk patients who may end up with a pressure injury?’ Some of our hospitals, including the main campus, are really large, and we needed a better way to prioritize, so that’s what we did. We partnered and co-designed a solution to help flag patients who are at highest risk for developing a pressure injury. We recently brought that product live.
Some of our best ideas have come from the frontline team. As leaders, it’s really important that we’re out there staying in touch and engaged, especially when a lot of things are done virtually. We need to make sure we make time and space for folks out there doing the work.
Gamble: Absolutely. Well, that’s about it for now. Thank you so much for your time. We really appreciated being able to hear your story and learn what your team is doing.
Freeman: Thanks, Kate. Anytime.
Share Your Thoughts
You must be logged in to post a comment.