How Does KLAS Conduct Research?

I can still remember when, as a kid, I discovered the fine print at the bottom of my cereal box announcing, “You must be 18 years of age to enter” as a caveat to their “win your own spy-gear tool kit” contest. I learned two things that day: first, becoming James Bond isn’t a solid career path, and second, always check for fine print.

In adulthood, that second lesson has proven invaluable. “Terms and conditions may apply” accompanies every contest, sweepstakes, and advertisement these days. Even the radio isn’t safe from the verbal equivalent of fine print as every commercial closes out with fast-spoken legalese.

KLAS is what you might call a large-print company in a fine-print world. As we seek to drive transparency in healthcare IT, we often act as the magnifying glass for the industry’s “see dealer for details” problems. It is important, then, for us to share some of the nitty-gritty details of how we gather the very research that promotes this transparency.

In a perfect world, several things would occur as KLAS conducted research:

• We’d have access to a full list of every organization and respondent who could or should give feedback.
• We’d obtain a 100% response rate from our randomly selected sample.
• All participants would be honest, and their feedback would be accurately captured.

While we don’t live in a perfect world, KLAS does take many steps to make sure our methodology runs as close to the ideal as possible.

Defining the Population

Right now, KLAS is actively tracking data for over 900 solutions in our database, which is updated daily. Given the immensity of the project, making sure that we have a solid sampling base for each product (one large enough to obtain a random sample) can be tricky.

Typically, our first solution is to work with the vendors themselves. We encourage them to share their full site lists with us, and we work to verify that the lists we receive are indeed complete. When we don’t have a complete site-list, our team of researchers spends time digging through user groups, seeking referrals, and reading press releases (among other methods) to uncover as complete a list as possible. KLAS often finds, during our 30,000 proactive provider interviews each year, customers using nearly every solution imaginable.

In all our interactions, we consider the relationships we build with providers as paramount. The strong relationships we form allow us to ask more than simple scoring questions. Instead, we're able to drill-down into what providers are doing that's new, what other products they're using and what trends they're seeing firsthand in the industry. This goes a long way toward helping us succeed, even when we're not able to work from a vendor's site list.

Taking a Random Sample

One of the biggest obstacles in obtaining a truly random sample is the fact that KLAS cannot control for a nonresponse bias when providers choose not to participate. We don’t hold any grudge against those who, for whatever reason, cannot engage with KLAS, and we still invite them to come and find insights from our data.

However, we do still need to overcome this bias as best we can. Our solution has been to try to build and strengthen relationships with the providers we meet. The better the relationship building of KLAS, the more likely that providers will become participants. It just so happens that we love working with the best and brightest in healthcare from across the world, so we call it a win-win.

It also helps that the senior leadership people KLAS interacts with often give feedback on a broad spectrum of solutions, helping to eliminate the nonresponse bias.

Conveying Accurate and Impartial Results

Of course, all of these measures amount to nothing if we’re ineffective at delivering these hard-gathered insights to the people who need them most.

I had a conversation just the other day with a provider in the middle of an EMR buying decision for her small practice. She told me that she relied heavily on our reports to guide her decision. She also mentioned that she was using the buying decision as part of a capstone project for her advanced IT degree. She assured me that she received an A+ for the assignment, at which I jokingly assured her KLAS would happily take some credit.

Our focus, after all this effort to gather the right data, is to ensure that the results we publish to the industry clearly and effectively communicate the voice of the provider and provide usable insights. For me, it’s sufficient to say that if KLAS had any “fine print,” it would say something to the effect of, “KLAS has no opinion. Only the data speaks.”

Keep an eye out in the few days, as our own Jenifer Gordon will be following up with how KLAS goes about handling this gathered data.