Intermountain Health saves thousands of hours, improves patient care with Azure OpenAI Service

Breathing new life into healthcare through AI observability and cloud infrastructure

Connecting 34 hospitals, 400 clinics, and more than 66,000 caregivers, Intermountain Health has a rich history of providing quality, sustainable, and affordable healthcare to millions of people. The Salt Lake City–based healthcare system supports patients and communities in Utah, Idaho, Nevada, Colorado, Montana, and Wyoming with best-in-class care at every stage of life. It has positioned technology, including AI and machine learning, at the forefront of its business model to streamline operations, distill data, scale patient communications, and achieve healthcare breakthroughs more efficiently.

In its pursuit to reimagine care, Intermountain prioritizes and champions responsible AI. For transparency, fairness, and reliability, responsible AI governance committees review both externally and internally developed AI solutions. Intermountain also expanded an enterprise AI and data science group, born out of its population health team, to accelerate the improvement of patient care with innovative AI and machine learning products.

AI observability is central to Intermountain’s commitments and is built into every stage of the healthcare pipeline. Having insight into the performance and fairness of AI products not only supports current and future planning and service reliability but also helps build trust with patients and clinical workers, making technology initiatives more transparent and effective. By implementing Arize Al for LLM evaluation and AI observability, Intermountain can proactively monitor and mitigate the impact of AI performance issues.

Inspired by the work of pioneering doctors in its network, including those who established early electronic medical records systems and clinical quality improvement programs, Intermountain is reducing caregiver burnout, streamlining clinical operations, and improving patient care every day. To sustain this success and develop scalable and reliable use cases, it wanted to combine AI, language models, and advanced algorithms with cloud infrastructure. By running its AI infrastructure on Microsoft Azure, Intermountain has done that.

Committing to trustworthy AI and delivering better performance, consistent patient outcomes

Intermountain first gained interest in AI and machine learning as potential solutions for complex, specialized health industry challenges, particularly patient outcome improvement and healthcare affordability. When Intermountain’s Digital Technology Services (DTS) caregivers started to develop, accelerate, and interweave the system’s cloud and AI strategies, the goal was to have caregivers use Azure Databricks and Arize, alongside Azure OpenAI Service and Azure API Management services, to deploy new, advanced, and custom AI models quickly and responsibly with clear observability. “You can deploy something very quickly but forget to enable machine learning observability. Suddenly you have a great product, but you don’t know its performance and whether it’s helping or harming patients,” explains Mark Nielsen, Senior Machine Learning Operations Engineer at Intermountain Health. “The cloud infrastructure that we built with Azure and Arize AI has been helpful in meeting both of those needs, and we’ve been able to speed up our deployment process while keeping our commitment to responsible AI.”

Always fine-tuning and improving, one of Intermountain’s goals is to have enough trust from caregivers that they start to reap the reward of improving patient outcomes, in addition to clinical caregivers spending less time at home documenting their visits. On the DTS side, engineers are already able to spin up or audit code quickly without feeling like they need to spend time outside work hours getting things done.

Using AI and cloud tools as a force multiplier in an innovation-led culture

With Azure OpenAI, Intermountain caregivers take advantage of new AI language models as they evolve and build or adapt custom models for maximum benefit. They use Azure Databricks and API Management to develop and deploy AI models and cloud resources with minimal administrative effort and very securely use industry-standard models at scale. “We have a culture of continuous improvement and mission-driven technical caregivers who help us maintain our level of innovation, efficiency, and excellence,” says Nielsen. “Easily scalable cloud tools become a force multiplier by making it simple to be agile, iterate, and improve.”  

Given its focus on custom model development, Intermountain set up automated continuous integration and continuous delivery processes using GitHub Actions for Azure to empower data scientists and machine learning engineers to make quick checks and help ensure a minimum level of accuracy as they do development work. “If we’re creating a model and want at least 80% accuracy and see it performs lower than that, we know we need to iterate and try to improve it in a different way,” explains Nielsen. “These automated checks help us implement responsible models.”

To further support its responsible AI pillars and progress, Intermountain brought its technology partner Arize on board by configuring and installing its AI observability and large language model evaluation solution in the Azure environment. On top of monitoring its Azure OpenAI models, Intermountain DTS caregivers use the Arize platform to help them identify the root causes of poor performance and bias to make proactive, versus solely reactive, updates.

Saving hours of work and improving efficiency

As Intermountain continues to enhance and expand its Azure environment, it’s saving time and effort while extending its clinical impact. “Making Microsoft 365 and Microsoft Copilot available to all our caregivers, whether they’re clinical or engineers, helped us save an estimated 4,300 hours of work last year,” says Nielsen—and these numbers continue to grow.

For doctors and caregivers, spending less time on documentation is a major win. They can use the saved time to focus on complex clinical problems and improving patient experiences. As its next step, Intermountain is currently collaborating with Microsoft and several other leading healthcare organizations on enabling citizen development of personalized AI agents to automate more complex manual processes. This includes applications that allow nurses to focus less on paperwork and more on patients.

Likewise, with the help of Azure and Arize AI, Intermountain DTS caregivers don’t have to build and manage AI workloads themselves, saving 40 hours of work per quarter for each AI product. “As you get multiple AI products into production, those savings really start to grow,” shares Nielsen. “AI models that previously took months to deploy can be deployed in weeks, and in some cases, days.”

Notably, doctors and data scientists are no longer expected to become experts in the other’s field of study. Additionally, before moving to the cloud, Intermountain didn’t have dedicated Linux administrators on hand who were familiar with the tools and software required for machine learning computation. When necessary, data scientists had to set up their own systems, infrastructure, and tools needed to deploy AI without formal training. “Moving to Azure allows us to use these AI systems, solutions, and cloud tools without needing to be a jack-of-all-trades,” explains Nielsen. “We’re now able to be more efficient because we can stay within these prebuilt frameworks, such as Azure Databricks, to spin compute resources up and down really quickly and worry about less about tool administration.”