Healthcare AI 2022
Proven Outcomes with Data Science Solutions

This report includes two types of data: (1) customer satisfaction ratings for several key performance metrics, and (2) case studies from individual organizations detailing their use cases, outcomes, and lessons learned. Summaries of the case studies are shared on the following pages. The complete case studies can be found in the full report.

ClosedLoop.ai Provides Top-Shelf Experience; Satisfaction with Health Catalyst Surges Following Increase in Prescriptive Guidance

Customers are highly satisfied with ClosedLoop.ai, who is used by both payer and provider organizations to improve risk scores, decrease readmissions, close care gaps, and reduce claims denials. Reported strengths include strong implementations and consultative partnering and guidance. The vendor excels at helping customers identify the root problems they want to solve and then find models that will deliver the desired results. Additionally, ClosedLoop.ai is highly involved in change management, sharing best practices, and helping organizations implement them; this leads to better physician adoption and thus better outcomes (see next chart). One potential hindrance is the need for technical resources within the customer organization to support the solution and monitor the models.

Customer satisfaction with Health Catalyst has jumped sharply over the last year as the company has improved at digging into customers’ data and providing prescriptive guidance as to where they should focus their AI efforts. Customers report that this guidance enables them to focus on the right populations and problems. They speak highly of the vendor’s expertise and willingness to help them achieve their goals. Newer AI customers in particular have experienced close partnering during implementation.

Ease-of-Use Challenges Can Hinder Outcomes for Cerner and Epic Customers

Though Cerner and Epic EMR users often turn to their enterprise vendors for AI capabilities, ease of use is a concern and can hinder outcomes. Epic customers can choose a limited number of prebuilt models from a broad library at no additional cost. These users represent the bulk of Epic’s AI customer base, and most use the models for clinical use cases. A growing number of customers are starting to also license Cognitive Computing Developer Platform, which allows for custom model deployment. Customers feel the prebuilt models are generally well integrated into their workflows. However, many report that testing these models and getting them to a state where they are delivering outcomes requires a significantly larger lift than expected. Epic’s customer support is typically a strength, and many respondents say it is easy to get ahold of the support resources. Feedback on training and proactive guidance is mixed—some describe the documentation as strong, and others feel it lacks needed information. Customers must pay for additional prebuilt models (beyond those that are free), leading 25% of respondents to mention nickel-and-diming as a frustration. Despite a few challenges, customers are optimistic about Epic’s direction and believe the vendor will help them achieve their goals.

Adoption of Cerner’s AI solutions has recently grown. Users of the prebuilt ML models (referred to as Managed ML Models) report more success than those who develop their own models (via HealtheDataLab). Regardless of which solution they are using, many respondents report difficulty getting the models up and running. Users of the prebuilt models are more likely to report guidance from Cerner on how to utilize the data, and they are also more likely to report achieving desired outcomes (e.g., reduced readmissions and improved care management). Customers that choose to build their own models struggle more and say they would benefit from additional training, documentation, and guidance to help them achieve desired results.

Case Study Summaries

To more deeply explore the current state of healthcare AI, KLAS interviewed two of each vendor’s deepest AI adopters regarding their AI use cases and the outcomes being achieved. Summaries of the case studies are shared below. The complete case studies can be found in the full report. These case studies are meant to both showcase specific vendor capabilities and encourage healthcare organizations that have not harnessed AI technology to consider how it could benefit their organizations.

Summary of case studies

^{Note: Jvion declined to provide candidates for case studies.}

Recommendations from Successful Organizations 

Preimplementation

• Analytics solutions are foundational to AI and often need to be implemented before predictive/prescriptive models.
• Identify the problem you are trying to solve. Then determine whether AI is the right solution for the problem.
• Set meaningful, clearly articulated goals for your use cases.
• An out-of-the-box solution is a great place to start, but the models will require examination before the predictions or prescriptive output can be implemented.
• For each use case, collaborate and get buy-in across teams before, during, and after the implementation; create an inclusive governance structure that represents all stakeholder groups and plan carefully for change management.

“I often see people who want to throw AI at everything, but it isn’t the right solution for every problem. We run about 10 models. We get a very high ROI from all of those models because we were very prescriptive in determining which scenarios we could use AI in to really deliver value. Good opportunities for AI are scenarios that require a lot of data. People should focus on use cases where they need AI to learn a lot of data and predict various conditions. Not everything is a good use case for AI.” —CIO

Implementation

• Don’t get hung up on perfecting the incoming data. The machine learning needs to happen with your data as it really is.
• Plan for the model testing to take more time and resources than expected.
• Don’t spend too much time trying to perfect a model. Models that are “good enough” can still drive significant outcomes.
• Figuring out how to operationalize the model (i.e., defining the intervention) often requires more work than the actual build.
• Foster buy-in and adoption by helping end users understand the models and how the data is generated.

“The biggest challenge with AI is operationalizing the models. We could create models left and right all day long, but they don’t mean anything until they are put into operation and can make a difference in decision-making.” —VP

Change Management & Long-Term Success 

• Drive long-term outcomes and prevent model drift by making sure your team is educated and ready to support the models, which will need continued iteration.
• For long-term success, integrate the models into processes and systems, especially the EMR.

“I would tell people to go in with a pretty open mind because there is more to learn than people think. People are probably better off going in with an open mind about the possibilities rather than demanding x, y, and z. Developing the models is a very iterative process.” —VP

About This Report

The data in this report comes from two sources: (1) case study interviews with two of each vendor’s deepest AI adopters and (2) KLAS performance data.

Case Study Interviews

To more deeply explore the current state of healthcare AI, we interviewed two of each vendor’s deepest AI adopters regarding their AI use cases and the outcomes being achieved. Helpful insights from each of these interviews can be found in the full report.

KLAS Performance Data

Each year, KLAS interviews thousands of healthcare professionals about the IT solutions and services their organizations use. For this report, interviews were conducted over the last 12 months using KLAS’ standard quantitative evaluation for healthcare software, which is composed of 16 numeric ratings questions and 4 yes/no questions, all weighted equally. Combined, the ratings for these questions make up the overall performance score, which is measured on a 100-point scale. The questions are organized into six customer experience pillars—culture, loyalty, operations, product, relationship, and value.

Some respondents choose not to answer particular questions, meaning the sample size for any given vendor or solution can change from question to question. When the number of unique organization responses for a particular question is less than 15, the score for that question is marked with an asterisk (*) or otherwise designated as “limited data.” If the sample size is less than 6, no score is shown. Note that when a vendor has a low number of reporting sites, the possibility exists for KLAS scores to change significantly as new surveys are collected.