Premium Reports
Contact KLAS
 Download Whitepaper    Zoom in charts

Preferences

   Bookmark

Related Series

Interoperability Summit 2020 White Paper
|
2020
Interoperability Cornerstone Summit 2017
|
2017

 End chart zoom
Interoperability Summit 2022 Interoperability Summit 2022
* A page refresh may be necessary to see the updated image
New

Interoperability Summit 2022

author - Coray Tate
Author
Coray Tate
author - Mollie Hunt
Author
Mollie Hunt
author - Paul Warburton
Author
Paul Warburton
 
May 10, 2022 | Read Time: 8  minutes

Current Time Inside Cache Tag Helper: 5/24/2022 3:57:23 PM and Model.reportId = 2885

Interoperability has been a focus in healthcare for years, yet progress toward any goal is difficult without consistent, impartial measurement. On February 7–8, 2022, KLAS held our second Interoperability Summit as part of ongoing efforts to comprehensively measure and validate EMR and other application vendors’ interoperability performance. While a previous summit had focused on creating a framework for interoperability vendor measurement (see below), the chief goal of the 2022 summit—attended by 93 leaders from across the healthcare industry—was threefold:

to see the full dataset.

if you don't have a login, getting started is easy.

Provider Organizations
Fill out our registration form to verify your organization.
HIT Companies
Contact a KLAS representative for plans and pricing.
HtmlReportContent Current Time Inside Cache Tag Helper: 5/24/2022 3:57:23 PM and Model.reportId= 2885 and Model.HtmlReportContent_LastWriteTimeUtcInTicks=637878184959672815

orange 1Finalize questions for a set of surveys, each customized to one of the following core technology segments: ambulatory/acute care EMR, population health management, patient engagement, clinical communications, home-based care EMR, and facility-based post–acute care EMR


orange 2Establish a methodology for measuring and displaying the results of the surveys


orange 3Identify other areas of interoperability that could benefit from measurement in the future

framework for interoperability vendor measurement

Question Sets for Core Technology Segments

Prior to the summit, KLAS worked with a provider steering committee—affectionately dubbed the Fab 4 (see page 6)—to outline the goals of the summit, create initial question sets for each core technology segment, and critique the proposed methodology for measurement.

At the summit, participants were broken into groups based on their area of expertise. Each group was tasked with reviewing the initial question set for a specific core technology segment, and after the summit, this feedback was used to create finalized question sets for all segments except patient engagement. The feedback group for that segment determined that one question set would not be sufficient to encompass the wide variety of technologies used for patient engagement, so instead multiple surveys will be created for different subsegments within patient engagement. These question sets will be finalized at a later date.

Key things to note about the question sets:

blue 1The questions center on key topics that summit participants feel will be most helpful in ascertaining the effectiveness of a vendor’s interoperability technology. They are not comprehensive of all questions that could or should be asked.


blue 2The questions measure generally agreed upon functionality/capabilities that all vendors should offer or should be working to provide in a specific technology segment.


blue 3As measurement, analysis, and our associated understanding evolve, it is expected that the questions will be modified and improved accordingly.

The finalized question sets for all core technology segments except patient engagement can be found in the index of this report.

Finalized Methodology for Calculating and Displaying Ratings

Before diving into how results from the interoperability question sets will be displayed, there are a few important factors to consider.

blue 1Feedback will be solicited from vendors’ leading customers and not from a randomized sample of customers. KLAS will also speak to fewer customers than our standard methodology requires because validating possible outcomes does not require talking to a large number of customers.


blue 2Ratings calculated from the question sets will not reflect vendors’ average customer experiences but rather validate the ideal—i.e., what achievements are possible for provider organizations that are committed to establishing the necessary technology and governance strategies to drive interoperability.


blue 3KLAS will continue to refine and improve this methodology as we learn from the initial process of collecting and calculating feedback.

Collecting Feedback from Customers

Summit participants determined that for solutions to be fully rated in the research, the following data collection criteria must be met: (1) the vendor must provide a list of 30 unique customers, and (2) KLAS must speak to and receive feedback from at least 10 of those unique customers. In order for the interoperability research to include as much emerging technology as possible, KLAS will use a different measurement standard for vendors that can’t supply a list of 30 unique customers. Vendors for whom KLAS speaks to 5–9 unique customers will be marked as having limited data.

During the summit, participants expressed concern about the depth of knowledge and broad end-user experience required of people who will respond to the interoperability question sets. To mitigate this concern, KLAS will gather perspectives on interoperability from individuals in operations, clinical, and information technology areas. (It should be noted that at some organizations, such as home-based care or ambulatory organizations, one individual may represent more than one of these areas.) Soliciting perspectives from these three areas will greatly improve the quality of feedback that KLAS receives.

For each question in the survey, respondents will be asked to rate on a Likert scale how much they agree with the statement or how satisfied they are with their vendor’s contribution to a specific use case or outcome. Each option on the Likert scale will receive a point value: 2 points for strongly agree, 1 point for agree, 0 points for indifferent, -1 point for disagree, and -2 points for strongly disagree. KLAS will add the final points together and convert those into star ratings based on the percentage of possible points earned.

Depending on the data KLAS receives through this process, KLAS may need to grade vendors on a curve to provide meaningful differentiation, and research methodology may need to be adjusted.

Displaying the Ratings

Based on the feedback from summit participants, the results of the interoperability question sets will be presented as star ratings, with a separate star given for each pillar in the interoperability framework (shown below as ratings, which will appear in future KLAS reports).

interoperability pillars

The ratings will be based on the percentage of points earned out of the total possible and will be displayed as follows:

0%–20%: no star     21%–40%: one-quarter star     41%–60%: half star     61%–80%: three-quarter star     81%–100%: full star

The only exception to the percentage-based breakouts is the Breadth of Adoption pillar. Vendors that meet both of the data collection criteria (a vendor-provided list of 30 unique customers and KLAS-validated feedback from at least 10 unique customers) will receive a full star for Breadth of Adoption. Those who meet only one of the criteria will receive no star for this pillar.

Remeasuring Vendors

KLAS will measure interoperability vendors on an ongoing basis to determine whether vendor delivery is improving. No specific timeline for remeasurement has been established, but it will likely take vendors a few years to deploy new technology for remeasurement. There may also be an opportunity for vendors who put out major releases to have only key areas of their technology remeasured.

Other Areas of Interoperability to Measure

In an effort to better identify and understand additional opportunities for future interoperability research, KLAS invited the summit attendees to fill out a pre-conference survey, which asked participants to select their top areas of focus for further interoperability measurement. The most common request was to measure provider organizations’ adoption readiness and actual adoption of interoperability technology. In prior research, KLAS has primarily focused on vendors’ efforts to drive interoperability forward, but much of the healthcare industry’s success with interoperability relies on provider organizations, who are ultimately responsible for what data is shared.

requested areas of future interoperability measurement chart

Other research areas that participants expressed interest in include payers’ willingness to share data, vendors’ ratings of each other’s willingness to share and provide access to data, and vendors’ efforts to enable provider organizations to better address information blocking—the last of which is included in the interoperability question sets (see page 7). Another potential area of measurement is the usability of interoperability for different end users; summit attendees specifically feel that physicians’ experiences could be used to determine how easily data can be shared and consumed. Summit attendees also want KLAS’ research to expand to other areas of technology that could be used for healthcare interoperability, such as AI, third-party data-sharing applications, and integration engines.

Once the initial interoperability measurement is completed, KLAS will have the resources to work on other potential areas of measurement for improving interoperability. The healthcare industry has an opportunity to collaborate and determine the best way to share data going forward by creating data-sharing standards. During the course of the summit, participants brought up many other topics related to digital health and governmental and regulatory changes. KLAS will continue to evaluate these topics as needed.

Where Do We Go from Here?

Looking ahead, KLAS will begin gathering the lists of 30 unique customer organizations from vendors to validate and collect data on these organizations’ experiences in the core technology segments (except patient engagement). Once data collection is complete, we will publish a report for each segment; publication of these reports is planned to begin at the end of 2022 and continue through 2023. After these initial reports, KLAS will revise the question sets and, if needed, reconvene the summit to determine next steps for measuring interoperability adoption and its impact on healthcare organizations.

timeline of interoperability framework and measurement

Summit Attendees

Fab 4

Michelle Dougherty, MA, RHIA
Senior Health Informatics Research Scientist
RTI International

Timothy R. Zoph
Industry Advisor

Steven R. Lane, MD, MPH, FAAFP
Clinical Informatics Director
Sutter Health

Terrence O’Malley, MD
Physician
Harvard Medical School

Technology/Services Providers

Allscripts
Amazon Web Services
Arcadia.io
athenahealth
Cerner
Change Healthcare
Consensus Cloud Solutions
Epic
Health Catalyst
Homecare Homebase
Get Well
Greenway Health
IBM Watson Health
Infor
Innovaccer
InterSystems
Kareo
Kno2
Lightbeam
Lyniate
MatrixCare
MATTER
MaxMD
MedAllies
MEDITECH
NextGen Healthcare
Orion Health
Phreesia
PointClickCare
ResMed (Brightree/MatrixCare)
Roche

Healthcare Organizations 

Baptist Health South Florida
Carle Health
CHIME
CommonWell Health Alliance
Defense Health Agency
eHealth Exchange
Harvard Medical School
HCA Healthcare
HealthLinc
Innovation Care Partners
Intermountain Healthcare
Kootenai Care Network
Novant Health
Ohio Living
Rocky Mountain Care
RTI International
Sangre de Cristo Community Care
Santa Cruz Health Information Organization
Siskin Hospital for Physical Rehabilitation
Sutter Health
SYNCRONYS
UC Davis Health
UHIN
University of Nebraska Medical Center
University of Pittsburg Medical Center
University of Utah Hospital
USMD
White Plains Hospital
WellMed Medical Management
WellSpan Health
White Plains Hospital

Current Time Inside Cache Tag Helper: 5/24/2022 3:57:23 PM and Model.reportId = 2885

orange 1Finalize questions for a set of surveys, each customized to one of the following core technology segments: ambulatory/acute care EMR, population health management, patient engagement, clinical communications, home-based care EMR, and facility-based post–acute care EMR


orange 2Establish a methodology for measuring and displaying the results of the surveys


orange 3Identify other areas of interoperability that could benefit from measurement in the future

framework for interoperability vendor measurement

Question Sets for Core Technology Segments

Prior to the summit, KLAS worked with a provider steering committee—affectionately dubbed the Fab 4 (see page 6)—to outline the goals of the summit, create initial question sets for each core technology segment, and critique the proposed methodology for measurement.

At the summit, participants were broken into groups based on their area of expertise. Each group was tasked with reviewing the initial question set for a specific core technology segment, and after the summit, this feedback was used to create finalized question sets for all segments except patient engagement. The feedback group for that segment determined that one question set would not be sufficient to encompass the wide variety of technologies used for patient engagement, so instead multiple surveys will be created for different subsegments within patient engagement. These question sets will be finalized at a later date.

Key things to note about the question sets:

blue 1The questions center on key topics that summit participants feel will be most helpful in ascertaining the effectiveness of a vendor’s interoperability technology. They are not comprehensive of all questions that could or should be asked.


blue 2The questions measure generally agreed upon functionality/capabilities that all vendors should offer or should be working to provide in a specific technology segment.


blue 3As measurement, analysis, and our associated understanding evolve, it is expected that the questions will be modified and improved accordingly.

The finalized question sets for all core technology segments except patient engagement can be found in the index of this report.

Finalized Methodology for Calculating and Displaying Ratings

Before diving into how results from the interoperability question sets will be displayed, there are a few important factors to consider.

blue 1Feedback will be solicited from vendors’ leading customers and not from a randomized sample of customers. KLAS will also speak to fewer customers than our standard methodology requires because validating possible outcomes does not require talking to a large number of customers.


blue 2Ratings calculated from the question sets will not reflect vendors’ average customer experiences but rather validate the ideal—i.e., what achievements are possible for provider organizations that are committed to establishing the necessary technology and governance strategies to drive interoperability.


blue 3KLAS will continue to refine and improve this methodology as we learn from the initial process of collecting and calculating feedback.

Collecting Feedback from Customers

Summit participants determined that for solutions to be fully rated in the research, the following data collection criteria must be met: (1) the vendor must provide a list of 30 unique customers, and (2) KLAS must speak to and receive feedback from at least 10 of those unique customers. In order for the interoperability research to include as much emerging technology as possible, KLAS will use a different measurement standard for vendors that can’t supply a list of 30 unique customers. Vendors for whom KLAS speaks to 5–9 unique customers will be marked as having limited data.

During the summit, participants expressed concern about the depth of knowledge and broad end-user experience required of people who will respond to the interoperability question sets. To mitigate this concern, KLAS will gather perspectives on interoperability from individuals in operations, clinical, and information technology areas. (It should be noted that at some organizations, such as home-based care or ambulatory organizations, one individual may represent more than one of these areas.) Soliciting perspectives from these three areas will greatly improve the quality of feedback that KLAS receives.

For each question in the survey, respondents will be asked to rate on a Likert scale how much they agree with the statement or how satisfied they are with their vendor’s contribution to a specific use case or outcome. Each option on the Likert scale will receive a point value: 2 points for strongly agree, 1 point for agree, 0 points for indifferent, -1 point for disagree, and -2 points for strongly disagree. KLAS will add the final points together and convert those into star ratings based on the percentage of possible points earned.

Depending on the data KLAS receives through this process, KLAS may need to grade vendors on a curve to provide meaningful differentiation, and research methodology may need to be adjusted.

Displaying the Ratings

Based on the feedback from summit participants, the results of the interoperability question sets will be presented as star ratings, with a separate star given for each pillar in the interoperability framework (shown below as ratings, which will appear in future KLAS reports).

interoperability pillars

The ratings will be based on the percentage of points earned out of the total possible and will be displayed as follows:

0%–20%: no star     21%–40%: one-quarter star     41%–60%: half star     61%–80%: three-quarter star     81%–100%: full star

The only exception to the percentage-based breakouts is the Breadth of Adoption pillar. Vendors that meet both of the data collection criteria (a vendor-provided list of 30 unique customers and KLAS-validated feedback from at least 10 unique customers) will receive a full star for Breadth of Adoption. Those who meet only one of the criteria will receive no star for this pillar.

Remeasuring Vendors

KLAS will measure interoperability vendors on an ongoing basis to determine whether vendor delivery is improving. No specific timeline for remeasurement has been established, but it will likely take vendors a few years to deploy new technology for remeasurement. There may also be an opportunity for vendors who put out major releases to have only key areas of their technology remeasured.

Other Areas of Interoperability to Measure

In an effort to better identify and understand additional opportunities for future interoperability research, KLAS invited the summit attendees to fill out a pre-conference survey, which asked participants to select their top areas of focus for further interoperability measurement. The most common request was to measure provider organizations’ adoption readiness and actual adoption of interoperability technology. In prior research, KLAS has primarily focused on vendors’ efforts to drive interoperability forward, but much of the healthcare industry’s success with interoperability relies on provider organizations, who are ultimately responsible for what data is shared.

requested areas of future interoperability measurement chart

Other research areas that participants expressed interest in include payers’ willingness to share data, vendors’ ratings of each other’s willingness to share and provide access to data, and vendors’ efforts to enable provider organizations to better address information blocking—the last of which is included in the interoperability question sets (see page 7). Another potential area of measurement is the usability of interoperability for different end users; summit attendees specifically feel that physicians’ experiences could be used to determine how easily data can be shared and consumed. Summit attendees also want KLAS’ research to expand to other areas of technology that could be used for healthcare interoperability, such as AI, third-party data-sharing applications, and integration engines.

Once the initial interoperability measurement is completed, KLAS will have the resources to work on other potential areas of measurement for improving interoperability. The healthcare industry has an opportunity to collaborate and determine the best way to share data going forward by creating data-sharing standards. During the course of the summit, participants brought up many other topics related to digital health and governmental and regulatory changes. KLAS will continue to evaluate these topics as needed.

Where Do We Go from Here?

Looking ahead, KLAS will begin gathering the lists of 30 unique customer organizations from vendors to validate and collect data on these organizations’ experiences in the core technology segments (except patient engagement). Once data collection is complete, we will publish a report for each segment; publication of these reports is planned to begin at the end of 2022 and continue through 2023. After these initial reports, KLAS will revise the question sets and, if needed, reconvene the summit to determine next steps for measuring interoperability adoption and its impact on healthcare organizations.

timeline of interoperability framework and measurement

Summit Attendees

Fab 4

Michelle Dougherty, MA, RHIA
Senior Health Informatics Research Scientist
RTI International

Timothy R. Zoph
Industry Advisor

Steven R. Lane, MD, MPH, FAAFP
Clinical Informatics Director
Sutter Health

Terrence O’Malley, MD
Physician
Harvard Medical School

Technology/Services Providers

Allscripts
Amazon Web Services
Arcadia.io
athenahealth
Cerner
Change Healthcare
Consensus Cloud Solutions
Epic
Health Catalyst
Homecare Homebase
Get Well
Greenway Health
IBM Watson Health
Infor
Innovaccer
InterSystems
Kareo
Kno2
Lightbeam
Lyniate
MatrixCare
MATTER
MaxMD
MedAllies
MEDITECH
NextGen Healthcare
Orion Health
Phreesia
PointClickCare
ResMed (Brightree/MatrixCare)
Roche

Healthcare Organizations 

Baptist Health South Florida
Carle Health
CHIME
CommonWell Health Alliance
Defense Health Agency
eHealth Exchange
Harvard Medical School
HCA Healthcare
HealthLinc
Innovation Care Partners
Intermountain Healthcare
Kootenai Care Network
Novant Health
Ohio Living
Rocky Mountain Care
RTI International
Sangre de Cristo Community Care
Santa Cruz Health Information Organization
Siskin Hospital for Physical Rehabilitation
Sutter Health
SYNCRONYS
UC Davis Health
UHIN
University of Nebraska Medical Center
University of Pittsburg Medical Center
University of Utah Hospital
USMD
White Plains Hospital
WellMed Medical Management
WellSpan Health
White Plains Hospital

author - Natalie Hopkins
Writer
Natalie Hopkins
author - Jessica Bonnett
Designer
Jessica Bonnett
author - Andrew Wright
Project Manager
Andrew Wright
 Download Whitepaper

This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the full retail price. Please see the KLAS DATA USE POLICY for information regarding use of this report. © 2022 KLAS Research, LLC. All Rights Reserved. NOTE: Performance scores may change significantly when including newly interviewed provider organizations, especially when added to a smaller sample size like in emerging markets with a small number of live clients. The findings presented are not meant to be conclusive data for an entire client base.