The KLAS Model

This all works because you're willing to give us feedback.

The KLAS Model diagram


How it works

Below you’ll find two columns detailing how KLAS conducts research. The left column contains an in-depth description of our methodology. If you’re short on time (or aren’t a statistician), you can read the right column for a quick summary of the text in simpler terms.

Detailed explanation

In other words

How does KLAS do their sampling?

If KLAS sampling could be perfect, the following would occur:

  • KLAS would have a full inventory of every live organization and every respondent who could/should give feedback.
  • A randomly sampled portion of the above population would be selected, with all selected individuals participating with KLAS.
  • Participants would be completely candid with their feedback, and all feedback would be accurately captured.
  • Participants would not be allowed to self-select into the KLAS sample.
  • KLAS would honestly, clearly, and accurately report all responses, including sensitive results.

The moment that any of these conditions are broken, the KLAS sample is not perfectly random. Unfortunately, the top three conditions are nearly impossible to achieve.

Because these conditions are so difficult to achieve, KLAS has never claimed a perfect sample or statistical significance.

While KLAS feels that it would be misleading to claim a statistically significant, truly random sample, KLAS makes significant efforts to obtain and honestly report on a sample that accurately reflects the sentiment of all customers.

KLAS puts forth significant effort to find the true story of what is happening with every vendor solution. We talk to customers until we feel the story take shape. KLAS does not claim a perfect sample or statistical significance but rather talks to a much larger group than a traditional reference check. It is no longer necessary for providers and payers to rely solely on calls with 2-3 customers from a vendor’s reference list.

How does KLAS define the population for a study?

In order to accurately sample within a customer base, KLAS must know the full customer base of every IT solution and service that it tracks. KLAS currently tracks around 900 different solutions in its database. With so many solutions being used in the vast number of healthcare organizations worldwide, it is very difficult to know which solutions and services have been sold to which organizations.

  • KLAS makes significant efforts to work with every vendor we track, with good success. Vendors are asked to confidentially share their customer lists. Vendors who share their customer base are asked whether they have shared a complete or partial list of customers.
  • While KLAS has strong, trusting relationships with many industry vendors, KLAS does not assume that a reported “full” client list from a vendor is actually a full client list. With over 30,000 interviews each year, KLAS often finds customers who have purchased a solution or services. This canvasing assists KLAS in understanding how forthcoming vendors are with revealing their complete client lists.
  • When a site list is not shared by the vendor or is not complete, KLAS makes efforts to find customers on our own. Online discussion boards, user groups, referrals from known customers, and press releases are a few of the ways in which customers are identified. In addition, KLAS has recently started tracking known software decisions with a focus on better identifying vendor customer bases.
  • KLAS also makes other efforts as necessary.

KLAS is actively tracking around 900 products. KLAS seeks to obtain a complete list of clients for every product. When a vendor won’t provide a full list, KLAS uses multiple methods to find as many customers to interview as possible.

Keys to make sure we don’t rely on vendors include: proactive calls to tens of thousands of healthcare executives, user groups, referrals, scouring the internet, and other useful methods.

How does KLAS gather a random sample?

While KLAS does randomly select within known customer bases regarding who we reach out to, KLAS cannot fully control for the nonresponse bias when customers do not participate.

KLAS relationships are critical to overcoming nonresponse bias within the KLAS sample. KLAS works to build long-term relationships with provider leadership so that a higher percentage of customers will participate in KLAS measurements (and because we sincerely enjoy working with incredible healthcare leaders from across the world). Providers who have participated with KLAS know that KLAS works diligently to provide valuable research to every participant.

KLAS has worked diligently to create a “KLAS community” where providers understand that their access to KLAS research is contingent upon their willingness to participate with KLAS when needed. Providers who access KLAS data are asked to “pledge” their willingness to participate in future KLAS studies when called upon.

In addition, senior healthcare leadership often gives feedback across a range of solutions, decreasing the correlation between nonresponse bias and customer satisfaction.

While KLAS can’t control whether a customer will participate, efforts are taken to build strong, mutually beneficial relationships with providers, leading to a higher response rate.

How does KLAS maintain data quality?

KLAS goes to great lengths to keep all feedback from participating providers strictly anonymous. As a part of KLAS research, provider verbatim quotes are captured, with identifying details carefully screened and removed in order to keep customer anonymity. KLAS’ track record of keeping customer anonymity for over 20 years gives participating providers confidence that they can speak freely and honestly.

In order to ensure that provider feedback is accurately captured, over 98% of all KLAS research is collected or validated over the phone by highly trained researchers. KLAS researchers must pass an extensive internal education course, which includes an hour-long verbal test administered by company executives. This intensive, six-month education program allows for KLAS researchers to be highly educated in the topics they are researching and greatly enhances the quality and accuracy of the data KLAS collects.

KLAS is commonly asked whether outlier responses are removed from the KLAS sample. Statistically, the process of truncating data (removing outlier responses) is done to ensure that possibly erroneous responses do not skew sample results. However, because KLAS researchers phone-validate over 98% of all responses, KLAS is confident that outlier responses are not a mistake but rather reflect very real customer sentiment that should be considered within the sample. It should be noted that all satisfaction scores below 60 (out of 100) and all perfect-100 scores require phone validation and undergo an additional level of scrutiny in order to ensure accuracy.

All provider feedback is kept strictly anonymous. KLAS also collects data via conversations (typically phone calls) to ensure data accuracy.

Researchers at KLAS go through (at minimum) an intense six month program to ensure they are able to do their jobs with excellence.

Exceptionally high or low scores go through additional scrutiny before the data is used/rejected.

How does KLAS control for self-selection sampling?

While providers are allowed to proactively give feedback on products of their choice online, this method of research makes up only a small portion of the research KLAS collects. However, even this small portion has the potential to skew KLAS research results, especially when a vendor has prompted their customers to participate with KLAS, knowingly or unknowingly skewing the KLAS sample.

Online surveys that are not the result of KLAS-selected sampling are phone validated and, under certain set parameters, set aside in a holding bin to compare against the KLAS-selected sample. Once enough holding bin data has been collected on a product to accurately compare the sample of the holding bin data against KLAS-selected research, statistical tests allow for KLAS to confidently identify whether the outside holding-bin data represents a random sample. Because KLAS does not control the sampling parameters of data coming in from online surveys, data whose distribution does not match the control KLAS sample is considered “in doubt” and is not included in the KLAS sample without verification from a trained researcher. It should be noted that the reason for not including this data in the KLAS sample is not that it represents an outlier opinion, but rather that KLAS cannot be confident in the data’s accuracy if it contradicts the KLAS-selected sample.

Data that comes from opt-in interviews (data taken without KLAS requesting it) is quarantined until additional bias checks, either from the researcher who took the call or the KLAS data team, have been run to confirm that these responses represent random-sample data.

Accurate, Honest, and Clear Communication of Results

Honesty is the most critical value that all KLAS employees must possess. KLAS is unapologetic in requiring the highest standards of honesty across the KLAS team. Despite having an incredible, trustworthy team, KLAS knows that our research must be as accurate as possible, so the KLAS data quality team (a group of non-researchers) monitors researcher calls and data.

While most industry vendors see the value that honest, accurate, and impartial research can provide to them and healthcare technology markets, on many occasions, vendors and outside groups have threatened KLAS and pressured them to hold back in sharing research and findings. Standing up to these outside pressures is not something that research methods can protect against, but rather requires a dedicated culture of loudly amplifying the provider voice, no matter the cost.

The KLAS research methodology is not perfect. At times, KLAS has identified mistakes we have made or mistakes that have been made in our sampling of customers and has reported those mistakes to the market. However, while mistakes do happen, KLAS team members believe that efforts to accurately measure customer satisfaction with healthcare technology, no matter how difficult, have an amazing ability to improve vendor delivery and the experiences of customers. We thank our incredible provider partners whose long hours with us on the phone and at conferences each year allow us to provide valuable measurements and insights to our provider customers.

KLAS employs a team of data specialists to do quality control on the data.

KLAS has, at various times, been pressured to change or suppress important insights. Our policy is always to present the truth, regardless of external forces.

KLAS will report as needed any mistakes that are made. We’re grateful to our payer and provider friends who are the foundation of our research.

Make a Difference

Join the KLAS Community to stand with other healthcare professionals by driving change within the industry.

Join Us