Reports

What Correlation Exists between Chart-Closure Rates and the Following Factors?

Net EHR Experience Score

Note: The Net EHR Experience Score (NEES) is a snapshot of providers’ overall satisfaction with the EHR. The survey asks respondents to rate factors such as the EHR’s efficiency, functionality, impact on care, and so on. The NEES is then calculated by subtracting the percent of negative user feedback from the percent of positive user feedback. Scores can range from -100% (all negative feedback) to +100% (all positive feedback).

What Is the Arch Collaborative Guidebook?

The Arch Collaborative Guidebook is the result of thousands of hours of combined effort. Over 250 health systems around the world have used the Arch Collaborative survey to measure the EHR satisfaction of their clinicians. With the survey taking roughly 10 minutes to complete, the 190,000+ clinicians who have participated have dedicated approximately 31,000 hours to providing feedback about their EHR satisfaction.

This incredible effort has allowed the Arch Collaborative to identify universal best practices that any health system—regardless of their EHR vendor—can use to improve their clinicians’ EHR satisfaction. These best practices are shared here to help healthcare leaders find their organization’s own path to EHR success. While the guidebook could be seen as a checklist, it will be most effective when its principles are skillfully customized to an organization’s specific culture and circumstances.

A Note about This Research

Though the Arch Collaborative’s primary focus is improving the EHR experience, the Arch Collaborative survey includes two questions about clinician burnout derived from the American Medical Association’s Mini Z study and correlated with the Maslach Burnout Inventory. While analyzing nurse responses to these questions can yield interesting insights, this report is not a comprehensive analysis of the causes of burnout, nor are its recommendations intended to be the primary solutions for this complex problem. Additionally, please note that most data in this report was collected prior to the COVID-19 pandemic. KLAS is actively measuring the impact of COVID-19 on burnout, and we hope to publish these findings at a future time.

EHR Satisfaction Is Not Strongly Correlated to Nurse Burnout

There is a slight, negative relationship between EHR satisfaction and nurse burnout, suggesting that as nurses become burned out, their EHR satisfaction declines. However, it is important to note that the quantitative relationship is quite weak—only about 5% of the variation in an organization’s Net EHR Experience Score can be attributed to burnout rate. So what factors do have a significant impact on burnout? Nurse respondents to the Arch Collaborative survey most commonly cite a chaotic work environment, time-consuming bureaucratic tasks, and a lack of effective teamwork. EHR-related contributors are among the least frequently selected. This suggests that the EHR as a whole may not be a significant factor in nurse burnout, and organizations should focus most burnout-reduction efforts on other areas.

Can EHR-Satisfaction Best Practices Also Help Reduce Burnout?

Despite the weak relationship between nurse burnout and EHR experience, the Arch Collaborative data has revealed that some practices that increase EHR satisfaction also lead to reduced burnout among nurses. For example, nurses who feel their documentation requirements are reasonable are less likely to be burned out. This suggests that reducing the documentation burden on nurses can help alleviate feelings of burnout. A case study from Peninsula Regional Medical Center demonstrates how the organization reduced their nursing documentation burden.

Another metric from the Arch Collaborative survey that impacts nurse burnout is how strongly nurses agree or disagree that their organizational leadership and IT department have implemented and supported the EHR well. This falls under the area of shared EHR ownership, where respondents are asked to rate three EHR stakeholders: themselves, their EHR vendor, and their organizational leadership/IT. Ratings for their organization’s leadership/IT are most closely correlated with a nurse respondent’s likelihood of being burned out—only 12% of nurses who strongly agree that leadership/IT delivers well experience feelings of burnout, well below the Collaborative average of 25%; conversely, 41% who strongly disagree report being burned out. Thus, organizations that focus on improving nursing trust in leadership/IT may see reduced levels of burnout. For best practices regarding leadership and IT, see “Organization Culture: The X-Factor of EHR Success.”

Key Takeaways & Further Research

Overall, the key takeaways for organizations to draw from this report are:

1. The majority of nurses do not appear to be experiencing burnout. While this is good news, there is still room for improvement for the 25% of nurses who do report symptoms.
2. There is only a small correlation between nurse burnout and EHR satisfaction, suggesting that organizations should focus most of their burnout-reduction efforts on other areas.
3. Decreasing nurses’ documentation burden and increasing their trust in organizational leadership/IT may help reduce their feelings of burnout.

For a broader examination of burnout across all types of clinicians, see KLAS’ 2019 report “Clinician Burnout.” Findings from that research reveal some best practices of health systems who experience lower-than-average burnout rates:

• Improve EHR proficiency through effective training
• Ensure documentation completion
• Give clinicians a voice in EHR governance
• Build strong teams
• Be deliberate about workloads
• Establish a culture where patients are truly the focus
• Strengthen communication skills for all caregivers
• Monitor and support improvements in team strength and culture

Learn More about the COVID-19 HIT Response

In response to the COVID-19 pandemic, KLAS is sharing timely insights on healthcare organizations’ experiences with their healthcare IT vendors and key solutions. Visit KLAS’ COVID-19 resource center to learn more and share your insights.

Telehealth: The Biggest Problem & Most Frequently Addressed—in the Short Term

A little under half of respondents (8 of 19) say telehealth functionality or capacity has been their number one problem to solve during the early stages of the pandemic. The good news is that nearly all respondents have a workable solution, and the rest are in the process of implementing one. Several mentioned using consumer-facing products—e.g., Apple FaceTime, Google Duo, Zoom, or Skype—that enable organizations without existing telehealth capabilities to quickly ramp up and handle increased demand. While many solutions are suitable for this emergency purpose, solutions that serve a strategic, long-term telehealth vision are much scarcer.

Detailed Executive Survey Results

A Note about This Research

These findings were gathered from organizations who chose to survey nurses as part of their EHR satisfaction measurement. The respondent sample size varies from question to question as not all organizations asked their nurses all of the standard survey questions and because the standard survey has periodically been updated with new questions. To increase the granularity of our nursing data, the Arch Collaborative survey was recently updated to gather information on nurse respondents’ clinical backgrounds, including clinical nurse specialists, LPNs, medical assistants, nursing assistants, RNs, and unit clerks. While breakouts by clinical background will be included in future reports, the change is too recent to be reflected in this current research.

What Factors Have the Greatest Impact on Nurse Satisfaction?

On average, nurses tend to report higher EHR satisfaction than physicians. Among nurses, what factors have the highest impact on EHR satisfaction?

What Are the Greatest Contributors to Nurse Burnout?

Which Areas of the EHR Are Most in Need of Optimization?

Many EHR functions need optimization to fit nurse workflows and can impede nurse efficiency, the delivery of care, or both if not implemented or trained on effectively. Nurses identified the following key EHR functions as most likely to negatively impact their efficiency or ability to deliver care.

Findings on the Three Pillars of EHR Satisfaction

Previously published Arch Collaborative reports illustrate how the Three Pillars relate to satisfaction:

†The Net EHR Experience Score (NEES) is a snapshot of clinicians’ overall satisfaction with the EHR environment(s) at the organization. The survey asks respondents to rate factors such as the EHR’s efficiency, functionality, impact on care, and so on. The Net EHR Experience Score is calculated by subtracting the percent of negative user feedback from the percent of positive user feedback. Net EHR Experience Scores can range from -100% (all negative feedback) to +100% (all positive feedback).

Key Finding #3:

Personalizing the EHR has a significant impact on satisfaction—in both the short term and the long term.

Personalization of the EHR can have a significant effect on EHR satisfaction, regardless of go-live timing. Organizations who went live prior to 2017 and those who went live in 2017 or later both have a difference in NEES of over 40 points between physicians with high personalization versus those with little or none. Some high-satisfaction Arch Collaborative members suggest giving physicians a strong foundation in EHR workflows and functions—and then introducing personalization to allow each user to tailor their EHR experience.

A Note about This Research

Currently, 103 hospitals that are ranked in at least one of the hospital quality programs listed above have also measured clinician EHR satisfaction via the Arch Collaborative survey (only 7 of the 103 are ranked in two programs). Given this small sample, the findings in this report must be considered preliminary.

Additionally, it is important to note that there are many metrics and programs by which hospitals can measure and benchmark their performance and quality. For this research, KLAS chose to focus on five of the most well-known ratings programs, each of which bases their results on different metrics and criteria.

On Average, Physicians at Ranked Hospitals Have Higher EHR Satisfaction

On average, physicians at organizations whose performance is celebrated in at least one of the key hospital quality programs examined in this report have higher EHR satisfaction than peers at non-ranked organizations. The relationship is weakest for organizations on the US News and World Report Best Hospitals list. While there is likely no causation in the relationship (i.e., it is unlikely that higher satisfaction with the EHR is a cause of high performance in these rankings, or vice versa), the correlation tells us that some high-performing hospitals also put strong emphasis on ensuring their clinicians achieve mastery of and high satisfaction with the HIT tools they use every day.

Nurse Experience with the EHR Follows the Physician Trend 

As with physicians, nurses who work at ranked facilities are more likely to be happier with their EHR than nurses at non-ranked organizations. There are two notable outliers to this trend. First, nurses at organizations that perform well in the Leapfrog ratings seem to be much happier with their EHR than nurses at non-ranked facilities (there is a nearly 27-point difference). Second, nurses at organizations in the CHIME Most Wired rankings are actually less satisfied on average than nurses at non-ranked organizations. This difference in satisfaction may be due to the small sample size of Arch Collaborative members that appear in the CHIME Most Wired rankings.

Other Arch Collaborative Resources Related to Quality

The case studies below provide additional quality-related learnings from Collaborative members.

• Anne Arundel Medical Center: “DataBay: Google-Like Capability”
• Anne Arundel Medical Center: “Integrating Data across Different EHRs”
• Novant Health : “Building Effective Best Practice Alerts”

Quantitative Data Trends Support the Voice of Clinicians

Quantitative measurements in Arch Collaborative data also back up what makes an EHR supportive (or not) of high-quality care. For example, some dissatisfied clinicians talk about alerts posing a threat to patient safety: as one said, the “high alert fatigue is frustrating and causes important messages to be overlooked.” Collaborative data backs this up—showing a strong relationship between the quality of alerts and how well the EHR enables high-quality care.

60% of the variation in organizations’ average quality-of-care rating can be explained by their helpfulness-of-alerts rating (R²=0.60). When alerts prevent mistakes in care delivery (rather than causing them), clinicians feel that their EHR enables high-quality care. To learn about an organization who has helped their clinicians set up meaningful alerts, read Novant Health’s case study “Building Effective Best Practice Alerts.” Alerts are just one example of a wider trend in KLAS data that lines up well with clinician commentary on what makes an EHR supportive of high-quality care. For more details on these findings, see the Expanded Insights section.

Promising Trends to Watch: 

Which Signal Metrics Show Preliminary Correlations with EHR Satisfaction and Burnout?

While Signal data is not designed to be a predictor of an individual physician’s EHR satisfaction or feelings of burnout, many high-performing Epic organizations in the Arch Collaborative have found success using their Signal data to spark conversations and guide interactions with physicians (see full report for examples). Additionally, several Signal metrics do show some degree of correlation with EHR satisfaction or burnout.

The Signal metric that currently does the best at predicting EHR satisfaction and burnout is the physician’s same-day appointment-closure rate. The more appointments physicians close the same day, the higher their EHR satisfaction and the less burnout they report. Presumably, those who close more appointments per day know the tool better and are managing their workload well. However, since individual physicians’ workflows vary dramatically, using this variable as a predictor may lead to false positives (i.e., physicians who appear to be struggling but are not).

Signal Data Not a Reliable Predictor of EHR Satisfaction or Burnout . . . Yet

Today, only about 4% of the variation in EHR satisfaction^† across individual ambulatory physicians can be predicted using Signal data (data set is limited). When we look at organization averages, this percentage increases but only to 15%. Signal data is slightly better at predicting physician burnout—5% of the variation in reported burnout between individual ambulatory physicians can be predicted using Signal data; 22% can be predicted across organizations.

†The Arch Collaborative uses the Net EHR Experience Score (NEES) to measure clinicians’ overall satisfaction with their EHR environment(s). Clinicians are asked to rate factors such as the EHR’s efficiency, functionality, impact on care, and so on, and the NEES is then calculated by subtracting the percent of negative user feedback from the percent of positive user feedback. The NEES can range from -100% (all negative feedback) to +100% (all positive feedback).

Actual vs. Predicted Net EHR Experience Score

Actual vs. Predicted Burnout Rating

Next Steps: We Need Your Help!

To date, comparing Signal usage data with EHR satisfaction data from the Arch Collaborative shows no reliable predictive correlation among ambulatory physicians. To further improve these findings, Epic and KLAS need more data points. If this research is interesting to you or your organization, please fill out this form to begin the process of adding your organization’s Signal data to our study.

54% of Organizations Are Seeing Statistically Significant Improvements

What Drove Improvements?

Rush University was the largest provider organization to see a dramatic improvement: a combination of repeat and new provider ratings brought the organization’s Net EHR Experience Score up by over 21 points (n = 36 repeats, n= 469 non-repeats; p <.001). Between their two surveys, Rush communicated to providers with, among other methods, the flyer below.

With this flyer, Rush University leadership focused on delivering improvements in the three pillars of EHR user optimization:

Summary

Provider organizations that think they cannot improve their EHR experience now have scientific data to rely on that demonstrates that improvement is not only possible, it is happening. It is within their reach to significantly improve on the clinicians’ experience. In order to drive improvements, organizations must first deliver an EHR foundation that is functional, stable, reliable, and responsive—and multiple EHR solutions are capable of being that type of foundation today. Then leaders must engage in user optimization efforts, focusing on aligning workflows through study and personalization, investing in user proficiency, and creating a sense of teamwork through shared EHR ownership.

For more insights, read the Expanded Insights section of this report, which dives deeper into these questions:

• What changes that organizations made were the most effective at driving change?
• What were the differences (if any) in improvement depending on organization type or size?
• What were the differences between provider and nurse satisfaction improvement efforts and outcomes?
• What types of organizations saw the greatest gains in satisfaction? Who was left behind?
• Which aspects of the EHR have seen the most improvement? What areas continue to lag?

Documentation Recipes

To determine how an organization’s documentation recipe is likely to impact EHR satisfaction and efficiency, each measured EHR deployment in the Arch Collaborative was sorted into one of eight groups depending on the organization’s average adoption level (as self-reported by physicians) of speech recognition, scribes, and dictation (adoption is considered high if it is above the median). While satisfaction and efficiency vary across these groups, not all variation is statistically significant. Overall, documentation method is a poor predictor of EHR experience and provider burnout (mathematically has only a limited impact). Statistically significant correlations are called out in future pages.

Shifting the Paradigm

Surprised by the lack of impact that documentation method has on EHR satisfaction and clinician burnout? KLAS encourages organizations to consider that how these technologies and services are implemented can play a significant role in how well they move the needle.

As noted above, the success of scribes is often dependent on the quality of the training and the expectations set for providers. Similarly, use of speech recognition technology today is not always accompanied by sufficient user education and workflow redesign. Though the technology is likely to change drastically in coming years (thanks to the development of ambient intelligence), KLAS asserts that proper training and user assistance will still be important success factors.

Outcomes by Recipe

Overall, the most effective organizations focus on only a couple of documentation methods (including personal entry and excluding dictation), likely because specializing in a couple of documentation methods allows for better support and training. Organizations with dictation or with many means of documentation see lower success.

Don’t Just Check the Box for Speech

Over 2,000 of the surveyed physicians commented on their speech recognition technology, with 45% reporting frustrations. The high number of positive comments indicates that the technology can be utilized well when accompanied by excellent EHR proficiency and speech recognition education. The quotes below are representative of the most common sentiments about speech recognition:

“I LOVE the new [version of our speech recognition solution]. I feel like after just a few days, it knows me better than the other version did after years of me yelling at it.”

“My biggest complaint is with [our speech recognition solution]. It’s not reliable. It shuts down and crashes Epic on a daily basis. I expected this to be resolved with the latest Epic upgrade, but it was not. I usually stop using [the speech recognition solution] for the day if it crashes. The cycle repeats itself the next day. When it works smoothly, it makes the workflow more efficient.”

“Please provide the opportunity for ongoing, hands-on, at-the-computer training for both [our speech recognition solution] and Epic. I believe giving employed physicians the opportunity to observe Epic-savvy staff use and optimize the system would be a great financial investment for the corporation. Additionally, having an optimization-minded advocate from our organization looking at user problems and resolving them with Epic would really reduce job frustration and improve user efficiency.”

The data shows that the use of speech recognition technology does not guarantee documentation success—an organization can achieve very strong or very weak results depending on how the technology is implemented and how well users are trained. With how speech recognition technology is commonly implemented today, providers often report better efficiency, similar or lower EHR satisfaction, and sometimes lower confidence in their own documentation. In general, physicians report better outcomes when organizations make an all-in bet with speech recognition (i.e., don’t use scribes or dictation).

Scribes

Among those using scribes, the differences are significant depending on how the data is analyzed.

The physicians who are given scribes are often the most frustrated, so it is not surprising that as a group, they report lower EHR satisfaction and worse chart-closure rates than their peers who use other documentation methods. However, the longitudinal data indicates that these metrics can improve with use of a scribe. Next year, KLAS will be able to show trends in how scribes impact physician burnout rates.

In Summary

It is increasingly clear that organizations looking for a quick fix to their EHR satisfaction challenges, either through scribes or speech recognition, are likely to be disappointed. Those that adopt these technologies and processes in tandem with strong change management and strong EHR proficiency will likely see efficiency gains.

It should be noted that the results achieved today could change as ambient intelligence supplements speech recognition technology, as organizations adopt better implementation and training programs for their technology environment, and as organizations improve how they implement scribe programs. As these technologies and services evolve, the Arch Collaborative will continue to monitor their impact and provide insights into how they can best be implemented and what results organizations can expect.

† Pediatrics and cardiology are cited here and in later paragraphs instead of other, more extreme examples due to their larger sample sizes and to avoid overemphasis on potential outliers in Collaborative data.

To determine how much of the physician EHR experience is reliant on training compared to the product itself, the following charts compare Cerner and Epic users (the two largest customer bases in Collaborative data), showing how training impacts their EHR experience relative to meeting specialty-specific needs.

Compared to Epic users, Cerner users tend to agree less that they have needed specialty-specific functionality, even when controlling for training quality. However, ophthalmology and dermatology stand out for lacking needed functionality regardless of EHR. This highlights EHR vendors’ responsibility to pay special attention to struggling specialties and ensure the EHR includes the functionality these specialties need to succeed.

While these findings may lead some less-satisfied physicians to believe that they can’t succeed with the EHR because of their specialty, trends within specialties debunk this assumption. In many cases, physicians in lower-scoring specialties who have had high-quality training report high overall EHR satisfaction. For example, cardiologists with strong initial EHR training have a Net EHR Experience Score of 58.3, well above the overall physician average. Conversely, physicians in higher-scoring specialties who receive poor training report low satisfaction; poorly trained pediatricians have a Net EHR Experience Score of -21.6.

While cardiology and pediatrics are vastly different specialties and use the EHR in different ways, this analysis
shows they are similarly susceptible to the positive effects of high-quality EHR training and the negative effects
of poor training.

A strong Net EHR Experience Score is correlated with lower burnout; the more satisfied providers in a certain organization are with their EHR, the less likely it is that many providers in that organization are experiencing burnout.

Guidebook Participants

KLAS is grateful for the dedication and participation of several leaders who have given their time and insights to ensure that the information in this guidebook is accurate and helpful. We commend and thank the following individuals for their guidance:

What Is the Arch Collaborative Guidebook?

The Arch Collaborative Guidebook is the result of thousands of hours of combined effort. Over 190 provider organizations around the world have measured the feedback of their clinicians, and the 100,000 clinicians who have participated by completing Collaborative surveys have dedicated an estimated 1,600 hours total to sharing their voices.

This incredible effort has allowed the Arch Collaborative to identify those organizations with the most successful EHR users. Leaders from these successful organizations have partnered with KLAS while also utilizing Arch Collaborative research findings to build this guidebook for EHR success.

There is no checklist for strong leadership; similarly, while this guide could be read as a checklist, it will be most effective when read as principles and opportunities for success meant to be skillfully applied according to an organization’s specific culture and circumstances.

This guidebook lays out the best practices identified in Collaborative data and shared by the most successful organizations in the Collaborative. If your organization is struggling with EHR satisfaction, start here!

It is expected that the principles in this guidebook will mostly remain consistent in future editions, but this guidebook may see significant additions over time as the Arch Collaborative continues to learn the science behind EHR success.

The principles in this guide come from two sources:

The biggest difference between US and non-US health systems is in adoption of EHR personalization tools. In the US, personalization tools help clinicians get data in and out of the EHR more quickly in the face of stringent documentation requirements. Outside the US, these requirements are less burdensome, so clinicians need less EHR personalization to be efficient. However, if the global community were to see increased regulatory burdens similar to those in the US, the use of EHR-personalization tools would likely have a significant impact on EHR satisfaction.