Arch Collaborative

Arch Collaborative data on EHR stakeholders shows that about 33% of the variation in the EHR experience from user to user can be attributed to the EHR vendor in use; the organization and the individual user account for the rest of the variation. While not the most important factor, the EHR vendor still has a big impact. EHR vendors have close relationships with their customer organizations, so they can apply needed solutions to improve end user satisfaction. Clinicians who strongly disagree that their EHR vendor delivers well are much less satisfied with the EHR experience overall than those who strongly agree (a difference in Net EHR Experience Score of 145.7 points—see below). EHR Vendor Programs Can Make a Big Difference To better understand the potential impact of the EHR vendor on end-user satisfaction, KLAS asked Cerner and Epic (the two vendors with the largest samples of Arch Collaborative member customers) to share what new tools, programs, or initiatives they have recently begun offering. The following descriptions come from the vendors and have not been validated by KLAS. Blueprint: Initiative by which Cerner works with clients to leverage the Blueprint tool to identify functionality that is owned but not yet implemented, fully configure existing capabilities to Cerner’s Model Experience, and provide guidance on adopting/using functionality in line with best practice recommendations. Brain: Inspired by what nurses commonly jot down on paper “nurse brains” at the start of their shifts, the Brain presents a timeline view of the orders, events, and requirements for each patient the user is assigned to. Hey Epic!: Voice assistant tool to help providers with documentation and information searches. Rover: A mobile app designed to improve productivity by allowing nurses and other clinicians to review, document, communicate, and complete other key workflows on the go. A number of organizations in the Arch Collaborative have recently implemented these tools or programs. The chart below compares these organizations’ Net EHR Experience Scores before and after implementing. Overall, these initiatives are correlated with higher EHR satisfaction. (It should be acknowledged that Collaborative members who re-measure tend to see some improvement in general, but the improvements shown below are still significant.) Overall, clinician feedback shows organizations who implement one of these initiatives and re-measure see an improved EHR experience. Though the extent of the impact can vary across initiatives and in specific metrics, and results are early, all initiatives highlighted in this report have a positive correlation overall with satisfaction improvement.

† As championed by Dr. CT Lin, CMIO at UCHealth, EHR optimization sprints aim to quickly optimize EHR efficiency through a three-pronged, team-based approach that includes (1) clinician training, (2) workflow redesign, and (3) the creation of specialty-specific tools. Dr. Lin—a longtime friend of the Collaborative and an early advisor to its members—graciously shared details of his EHR 2.0 optimization sprints at the Eastern US Arch Collaborative Workshop, held in January 2020. Additional details about UCHealth’s sprints can be found in this article by Mayo Clinic. Across Clinical Roles, Optimization Sprints Dramatically Improve EHR Satisfaction On average, providers, nurses, and allied health professionals all saw large increases in their Net EHR Experience Score (NEES) ‡ following an optimization sprint, and each group also reported improved satisfaction with how their organization has implemented, supported, and trained on the EHR. As IT and operational departments invest in reaching out to their clinics, they can dramatically change how clinicians view the EHR. ‡ Each individual clinician’s responses to the Arch Collaborative EHR Experience Survey regarding core factors such as the EHR’s efficiency, functionality, impact on care, and so on are aggregated into an overall Net EHR Experience Score (NEES), which represents a snapshot of the clinician’s overall satisfaction with the EHR environment at their organization. The NEES is calculated by subtracting the percent of negative user feedback from the percent of positive user feedback. An NEES can range from -100 (all negative feedback) to 100 (all positive feedback). Increased Efficiency ? Decreased Burnout for Providers On average, the percentage of providers reporting at least some degree of burnout dropped by nine percentage points following an optimization sprint. Several factors likely contribute to this reduction in burnout. However, of particular note is that pre-intervention, less than half of participating providers viewed their EHR as a tool that enables efficiency; post-intervention, that number rose to almost two-thirds, with several organizations reporting a dramatic reduction in afterhours charting. Provider perceptions of the efficacy of their ongoing EHR training also saw a significant boost. Other Collaborative data has shown that providers who don’t agree that their ongoing training is sufficient are 3.0–4.5 times more likely to report plans to leave their organization within two years. Framework for Implementing a Successful EHR Optimization Sprint While the Arch Collaborative organizations that have completed an optimization sprint offered a variety of helpful advice for their peers, the following two overarching guidelines were mentioned by all four organizations: Leadership matters —The individuals chosen to lead the sprint and the specific clinics chosen for the pilot will make or break the project. Look for leaders who are excited to champion operational improvements and increased clinician engagement. Utilize a pilot and then grow organically as you achieve success —For the organizations in this report, optimization sprints have become a key element of their efforts to support the EHR. However, this broad investment was only made after pilot programs demonstrated that sprints were an effective way to improve satisfaction with the EHR. Steps of a Successful Sprint Step 1: Build your sprint team Team members should be highly competent in either training clinicians or understanding EHR workflows. These individuals should also be generous in how they work with others, as they will likely be working with many clinicians who have very negative perceptions of the IT team and the EHR. Step 2: Choose your pilot clinic Start with a clinic that has a pressing need to improve the EHR and whose leadership is already bought into the idea of a sprint. Step 3: Prepare for the sprint The preparation phase typically includes a kickoff meeting followed by several additional meetings to identify the build scope and establish relationships between the sprint team and the clinic. These preparations are key to being able to hit the ground running once the sprint team is on-site at the clinic. Step 4: Collect a pre-sprint measurement Several organizations have utilized their EHR’s native tracking of metrics like afterhours charting and chart-closure rates to set a pre-sprint benchmark. Additionally, three of the four Collaborative members in this research established a benchmark utilizing the Collaborative’s pre/post surveys, which are included in all Collaborative memberships (alternatively, a recently completed full Collaborative measurement can be used as the pre-sprint benchmark). Step 5: Complete the sprint Sprints generally have three components: clinician training, workflow redesign, and the creation of specialty-specific tools. The training is typically done one on one and focuses on helping clinicians become proficient with new and pre-existing functionality and tools as well as any newly developed workflows. For specific details on how to implement a sprint, please see this comprehensive article in the Mayo Clinic Proceedings . Step 6: Collect a post-sprint measurement Once the sprint is complete, use the same survey instrument leveraged for the pre-sprint measurement to capture any changes in clinician satisfaction. Step 7: Adjust and iterate Using feedback from clinician surveys and interviews with clinic leadership to identify any needed adjustments, select the next clinic to participate in the sprint.

Program Goals As part of the organization’s digital transformation strategy, continually improve the clinical user experience of the EHR. Organizational Outcomes Scored in the 91st percentile for Net EHR Experience Score in 2020 Increased Net EHR Experience Score by 14 points for physicians Is a top-performing Cerner organization in the Arch Collaborative Collaborative-Verified Best Practices Develop a comprehensive change management strategy so all stakeholders are engaged and informed throughout Have the governance and steering committees engage key champions to help drive deliverables, buy-in, and user satisfaction/experience What KFSH&RC Did

33% of Repeat Individuals Report Significant Improvement Changes in Net EHR Experience Score (NEES) † between a clinician’s first and second measurement have an almost perfect normal distribution, validating the accuracy of the survey in gauging EHR satisfaction. While the average clinician reports a 6.3-point improvement, approximately 33% of clinicians have seen highly significant improvement (i.e., an increase in NEES of at least 20 points). Analyzing the data from these “high-improvement” clinicians reveals the factors most commonly associated with improved EHR satisfaction (see below). † Each individual clinician’s responses to the Arch Collaborative EHR Experience Survey regarding core factors such as the EHR’s efficiency, functionality, impact on care, and so on are aggregated into an overall Net EHR Experience Score (NEES), which represents a snapshot of the clinician’s overall satisfaction with the EHR environment at their organization. The NEES is calculated by subtracting the percent of negative user feedback from the percent of positive user feedback. An NEES can range from -100 (all negative feedback) to 100 (all positive feedback). The Individual Metrics Most Commonly Associated to High Overall Improvement Almost all clinicians in the high-improvement group report an increase in their personal accountability for mastering the EHR. This validates a key finding of the Collaborative, which is that—across metrics—users themselves can account for up to 40%–60% of the variation in their EHR satisfaction (see “Stakeholder” chart below). See the Intermountain Healthcare case study to learn how that organization’s focus on provider coaching led to improved perceptions of the EHR. Other common associated improvements include improved perceptions of the EHR vendor, increased satisfaction with the quality and frequency of ongoing training, and greater adoption of personalization. This validates the importance of two of the Arch Collaborative’s three pillars of EHR satisfaction: training (both initial and ongoing) is an important aspect of the user mastery pillar, while personalization is critical to the EHR “meeting unique user needs.” See full report for additional details on the three pillars. The Individual Metrics with Most Impactful Association to High Overall Improvement While personal accountability is the most common change, system integration is the core NEES metric associated with the largest increases in overall EHR satisfaction and represents a significant opportunity for organizations and vendors to completely shift their end users’ EHR perceptions. Regarding integration, respondents are asked to rate—on a five-point Likert scale—their level of agreement that the internal and external integration is sufficient. On average, respondents who increase by four levels of agreement (i.e., move from strongly disagree to strongly agree) see a corresponding NEES increase of 130.7 points. Even more modest shifts of one or two levels of agreement can still yield significant improvements. It is important to note that the core NEES metrics that are associated with the largest improvements in overall satisfaction—including integration as well as system speed and functionality—are actually the ones that individual clinicians have the least control over (see “Stakeholder” chart above). This highlights the fact that true EHR success does not depend on the user alone—it also requires collaboration from the vendor and organization to make meaningful changes that individuals cannot make themselves. Among the survey questions that don’t factor into a respondent’s NEES, the ones with the largest associated increases are the same metrics noted above as the most common—personal accountability, agreement that EHR vendor delivers well, and satisfaction with ongoing training. The strong association that exists between a respondent’s overall EHR satisfaction and their approval of the EHR vendor’s delivery again highlights the importance of provider organizations working closely with their vendors to hold them accountable for needed changes. Key Takeaways Changes across measurements show that a delicate balance must be achieved between three stakeholders: the individual, the provider organization, and the EHR vendor. When the three work together, it is possible to achieve significant improvement in EHR satisfaction. Individuals have the opportunity to improve their own satisfaction with the EHR by attending trainings, using tip sheets, and adopting more personalizations. Provider organizations and EHR vendors are responsible for making needed improvements to things such as integration, functionality, and system response time. Efforts in these areas can compound end-user efforts and transform EHR detractors into evangelists.

Program Goals Identify where problems exist within the EPR Provide each ward with access to the IT team Develop clinician EPR design expertise to help drive improvements Organizational Outcomes 15-point Net EPR Experience Score (NEES) improvement for doctors 20% increase in agreement that ongoing training is sufficient for doctors 14-point NEES improvement for nurses 16% increase in agreement that ongoing training is sufficient for nurses Collaborative-Verified Best Practices Round frequently to provide the IT team with key understanding of the needs of end users Collate up-to-date list of IT/hardware/Wi-Fi issues and promptly address and fix them to improve the experience for end users Focus on end-user mastery, both in providing easier access to help and placing some of the responsibility of success on groups of end users What University Hospitals of Morecambe Bay Did

Program Goals Decrease provider turnover rate Get rid of outdated or inefficient workflows Increase EHR personalization Enhance trust in organization leadership/IT Organizational Outcomes 95th percentile for provider Net EHR Experience Score 94th percentile for provider agreement that EHR vendor delivers well 89th percentile for physician agreement that initial training is sufficient 94th percentile for physician agreement that ongoing training is enough 24% improvement in agreement that initial training is sufficient from 2019 to 2021 for physicians 30% improvement in agreement that ongoing training is enough from 2019 to 2021 for physicians Collaborative-Verified Best Practices Initial EHR education trains on organization-specific workflows Ongoing EHR education includes many different methods, including one-on-one support and virtual training Consistent rounding has created good relationships between providers and their trainers Multidisciplinary governance committees create a fast track for EHR adjustments Providers are offered protected time to set up their personalization within the EHR What The Guthrie Clinic Did

This report, an update to the 2019 Clinician Training report , dives into evidence-based findings on EHR training. It provides a refresh on past findings and shares insights from new questions (on telehealth, virtual training, etc.) and new views of Collaborative data. Ultimately, this report aims to help organizations elevate training to improve clinician EHR satisfaction, clinician wellness, and the quality of patient care. (See the full report for the complete insights on clinician training.) Keys to Successful EHR Training More than 20 organizations have participated in the Trainer Quality Benchmark survey, which collects responses from clinicians after they receive EHR training. Data from this survey reveals two aspects of training are highly correlated with satisfaction: type of training and length of training. Various types of training can be effective as long as an actual trainer is involved—self-directed e-Learning is much less effective. More than an hour of training is also likely to result in higher training satisfaction. Initial Training Has Consistently High Correlation with Satisfaction Clinicians who strongly agree that their initial EHR training prepared them well to use the EHR have an average Net EHR Experience Score (NEES) 89.7 points higher (on a -100 to 100 scale) than those who strongly disagree. This is the exact same spread reported in the 2019 Clinician Training report , even with 50,000 additional responses collected since then. (More insights on initial training can be found in the Expanded Insights and on the Arch Collaborative website in the form of webinars, case studies, and other reports.) Early Insights on the Use of Simulations A new question in the executive survey (conducted with executive leaders at member healthcare organizations) asks whether the organization uses simulations for initial EHR training. Preliminary results show that organizations that do use simulations have, on average, a higher NEES than organizations that don’t. ‡ The Net EHR Experience Score (NEES) is a snapshot of clinicians’ overall satisfaction with the EHR environment(s) at the organization. The survey asks respondents to rate factors such as the EHR’s efficiency, functionality, impact on care, and so on. The Net EHR Experience Score is calculated by subtracting the percent of negative user feedback from the percent of positive user feedback. Net EHR Experience Scores can range from -100 (all negative feedback) to +100 (all positive feedback). Strong Ongoing Training Associated with 100-Point Higher Satisfaction Arch Collaborative data shows a 101.2-point difference in NEES between clinicians who strongly agree ongoing training is sufficient and those who strongly disagree. KLAS’ 2019 report found a 102.7-point difference. The static nature of these results indicates the consistent importance of training satisfaction, even as Collaborative data has expanded to include more and more organizations. Telehealth Training and the EHR Experience In the last year, KLAS has added a question to our EHR Experience Survey about training on telehealth tools. Responses to this question show strong training on telehealth tools and processes is correlated with a better EHR experience overall. For deeper insights on the effects of telehealth training, see the Expanded Insights.

Program Goals Develop provider mastery of the EHR across the health system Organizational Outcomes Improve provider Net EHR Experience score (NEES) by 33 points in one year Collaborative-Verified Best Practices Focus on enabling clinicians to develop EHR mastery Advertise the coaching opportunities clearly so that many providers have a chance to participate Follow up after initial training to allow for clinicians to understand where they may need more education and come back several times to train to enable this to take place What Intermountain Healthcare Did

59% of Organizations Have Seen Statistically Significant Improvement What Drives Improvement?  An Intermountain Healthcare Case Study Intermountain Healthcare’s first Arch Collaborative measurement (in June 2019) showed widespread dissatisfaction with the EHR. Seeking to improve the clinician experience, Intermountain invited KLAS and four high-performing Collaborative members (Kaiser Permanente Northwest, Memorial Health System, Sharp HealthCare, and UCLA Health) to an on-site “peer guidance” program. As a result of the insights gained, Intermountain launched a series of new training efforts, including flexible, individualized EHR coaching for providers. Early efforts targeted 87 family and internal medicine providers, and Intermountain used the Collaborative’s pre- and post-surveys to measure the training’s impact. This pilot revealed that a one-hour session saved providers a perceived 63 minutes per week and increased their NEES by an average of 40 points. Following this early success, Intermountain shifted 15 FTEs from design, development, and support areas to coaching and mastery. By the end of 2020, providers across Intermountain reported an average NEES 33 points higher than at Intermountain’s original measurement. Newly hired providers now receive a one-week follow-up session and four to six optimization sessions, and Intermountain continues to develop various training modalities—including webinars and short videos—to expand access. Intermountain utilizes the following strategies to boost participation among providers: Enlists clinical leaders, EHR champions, and division chiefs to promote the coaching Emphasizes that coaching leads to less time in the EHR as well as improved clinician wellness (i.e., “there is no downside to doing this”) Highlights that benefiting from coaching is universal, not atypical (using this Atul Gawande article in The New Yorker as a talking point) Makes it easy to attend: User chooses the time/location of the coaching (either in person or virtual) Direct sign-up links sent via email Screen savers include a QR code for signing up from personal smartphones (no login required) Continued Improvement beyond Second Measurement Several Collaborative members have now measured EHR satisfaction organization-wide three or more times and have continued to see improvement in their clinicians’ EHR experience (including among their least satisfied users). Franciscan Health has done four organization-wide measurements. After their first measurement (2017) revealed the need to improve clinician satisfaction with the EHR, Franciscan reorganized their governance structure to improve operational ownership of EHR improvement. Following their 2019 follow-up measurement, the organization centralized their training program and began using the Collaborative’s trainer quality benchmark survey to measure trainer quality. In 2020 (even in the midst of the pandemic), a majority of their providers completed an EHR refresher course. Today, Franciscan is working to improve onboarding for nurses and make sure each provider receives one-on-one attention. (More details on Franciscan’s improvement efforts can be found here .) Patterns of Success Of the 46 organizations that have measured EHR satisfaction organization-wide at least twice, 36 have also filled out an executive survey describing what changes they have made as a result of their measurement and what impact these changes are having. 17 of these organizations have achieved statistically significant improvements (increase of at least 8 points) between their two most recent measurements. The most common pathways of success reported by these organizations were identified via a qualitative comparative analysis and are summarized in the graphic below. Though not as commonly implemented as the above strategies, improving EHR governance actually has one of the biggest positive impacts on EHR satisfaction. Of the 36 organizations that filled out the executive survey, 7 report implementing major changes to their EHR governance, and 4 of these have seen statistically significant satisfaction increases among repeat respondents. More details on this and the other findings noted above can be found in the Expanded Insights section of the report. Key Takeaways No matter what EHR satisfaction is like at your organization today, efforts to improve will likely yield results. Improvement is an iterative process—initial satisfaction measurements identify areas for improvement, and repeat measurements can help gauge the efficacy of implemented interventions. Going big (e.g., making major changes to EHR governance or retraining a significant portion of clinicians) isn’t without risk but will likely be successful and yield the largest rewards. Efforts to improve the EHR experience are likely to also improve efficiency and the EHR’s effect on quality of care.

Program Goals Gather feedback from all end users to support optimal EHR enhancements for all involved parties Create an effective communication strategy around all EHR changes Organizational Outcomes Nurses score in 90th percentile for Net EHR Experience Score Collaborative-Verified Best Practices Clinician Relationships and Communication Nursing EHR Success Shared Ownership and Governance What Bellin Health Did

Program Goals Train nurses to use the EHR well so that they can be prepared when they see their first patient. Organizational Outcomes 88th percentile for nurse agreement that initial EHR training prepared them well 89th percentile for nurse EHR satisfaction 39% improvement in nurses’ agreement that initial EHR training prepared them well between their 2017 and 2020 measurements Collaborative-Verified Best Practices Nursing EHR Success Onboarding EHR Education What Franciscan Health Did

Program Goals Build a working relationship between clinicians and IT based on mutual respect and trust Organizational Outcomes 95th percentile for nursing trust in IT Collaborative-Verified Best Practices Recognized Improvement Clinician Relationships and Communication What TidalHealth Did