The KLAS Model

How does KLAS define the population for a study?

In order to accurately sample within a customer base, KLAS must know the full customer base of every IT solution and service that it tracks. KLAS currently tracks around 900 different solutions in its database. With so many solutions being used in the vast number of healthcare organizations worldwide, it is very difficult to know which solutions and services have been sold to which organizations.

• KLAS makes significant efforts to work with every vendor we track, with good success. Vendors are asked to confidentially share their customer lists. Vendors who share their customer base are asked whether they have shared a complete or partial list of customers.
• While KLAS has strong, trusting relationships with many industry vendors, KLAS does not assume that a reported “full” client list from a vendor is actually a full client list. With over 30,000 interviews each year, KLAS often finds customers who have purchased a solution or services. This canvasing assists KLAS in understanding how forthcoming vendors are with revealing their complete client lists.
• When a site list is not shared by the vendor or is not complete, KLAS makes efforts to find customers on our own. Online discussion boards, user groups, referrals from known customers, and press releases are a few of the ways in which customers are identified. In addition, KLAS has recently started tracking known software decisions with a focus on better identifying vendor customer bases.
• KLAS also makes other efforts as necessary.

How does KLAS gather a random sample?

While KLAS does randomly select within known customer bases regarding who we reach out to, KLAS cannot fully control for the nonresponse bias when customers do not participate.

KLAS relationships are critical to overcoming nonresponse bias within the KLAS sample. KLAS works to build long-term relationships with provider leadership so that a higher percentage of customers will participate in KLAS measurements (and because we sincerely enjoy working with incredible healthcare leaders from across the world). Providers who have participated with KLAS know that KLAS works diligently to provide valuable research to every participant.

KLAS has worked diligently to create a “KLAS community” where providers understand that their access to KLAS research is contingent upon their willingness to participate with KLAS when needed. Providers who access KLAS data are asked to “pledge” their willingness to participate in future KLAS studies when called upon.

In addition, senior healthcare leadership often gives feedback across a range of solutions, decreasing the correlation between nonresponse bias and customer satisfaction.

How does KLAS maintain data quality?

KLAS goes to great lengths to keep all feedback from participating providers strictly anonymous. As a part of KLAS research, provider verbatim quotes are captured, with identifying details carefully screened and removed in order to keep customer anonymity. KLAS’ track record of keeping customer anonymity for over 20 years gives participating providers confidence that they can speak freely and honestly.

In order to ensure that provider feedback is accurately captured, over 98% of all KLAS research is collected or validated over the phone by highly trained researchers. KLAS researchers must pass an extensive internal education course, which includes an hour-long verbal test administered by company executives. This intensive, six-month education program allows for KLAS researchers to be highly educated in the topics they are researching and greatly enhances the quality and accuracy of the data KLAS collects.

KLAS is commonly asked whether outlier responses are removed from the KLAS sample. Statistically, the process of truncating data (removing outlier responses) is done to ensure that possibly erroneous responses do not skew sample results. However, because KLAS researchers phone-validate over 98% of all responses, KLAS is confident that outlier responses are not a mistake but rather reflect very real customer sentiment that should be considered within the sample. It should be noted that all satisfaction scores below 60 (out of 100) and all perfect-100 scores require phone validation and undergo an additional level of scrutiny in order to ensure accuracy.

How does KLAS control for self-selection sampling?

While providers are allowed to proactively give feedback on products of their choice online, this method of research makes up only a small portion of the research KLAS collects. However, even this small portion has the potential to skew KLAS research results, especially when a vendor has prompted their customers to participate with KLAS, knowingly or unknowingly skewing the KLAS sample.

Online surveys that are not the result of KLAS-selected sampling are phone validated and, under certain set parameters, set aside in a holding bin to compare against the KLAS-selected sample. Once enough holding bin data has been collected on a product to accurately compare the sample of the holding bin data against KLAS-selected research, statistical tests allow for KLAS to confidently identify whether the outside holding-bin data represents a random sample. Because KLAS does not control the sampling parameters of data coming in from online surveys, data whose distribution does not match the control KLAS sample is considered “in doubt” and is not included in the KLAS sample without verification from a trained researcher. It should be noted that the reason for not including this data in the KLAS sample is not that it represents an outlier opinion, but rather that KLAS cannot be confident in the data’s accuracy if it contradicts the KLAS-selected sample.

Accurate, Honest, and Clear Communication of Results

Honesty is the most critical value that all KLAS employees must possess. KLAS is unapologetic in requiring the highest standards of honesty across the KLAS team. Despite having an incredible, trustworthy team, KLAS knows that our research must be as accurate as possible, so the KLAS data quality team (a group of non-researchers) monitors researcher calls and data.

While most industry vendors see the value that honest, accurate, and impartial research can provide to them and healthcare technology markets, on many occasions, vendors and outside groups have threatened KLAS and pressured them to hold back in sharing research and findings. Standing up to these outside pressures is not something that research methods can protect against, but rather requires a dedicated culture of loudly amplifying the provider voice, no matter the cost.

The KLAS research methodology is not perfect. At times, KLAS has identified mistakes we have made or mistakes that have been made in our sampling of customers and has reported those mistakes to the market. However, while mistakes do happen, KLAS team members believe that efforts to accurately measure customer satisfaction with healthcare technology, no matter how difficult, have an amazing ability to improve vendor delivery and the experiences of customers. We thank our incredible provider partners whose long hours with us on the phone and at conferences each year allow us to provide valuable measurements and insights to our provider customers.