An Exclusive Interview with Adam Gale, EVP of Operations, KLAS

After some spirited KLAS discussion by HIStalk readers, I e-mailed Adam Gale and suggested that the company's best response might be to interview with HIStalk (after some encouragement by a provider-side colleague who also reads here.) Certainly few of us, even those who have contributed to or used KLAS reports over the years, have an understanding of how their business works. I was happy when he agreed.

KLAS is an online subscription database containing results of surveys software application users. Those responses are compiled into a rating system segregated by product. If you look up radiology systems, for example, you'll see them sorted in order of user rating, along with considerable detail about how each product did in individual categories (fairness of contract, support, and the all-important "would you buy it again.") For something that important, it's not surprising that readers from both vendor and provider organizations are curious about how KLAS works and how the numbers are derived.

Thanks to Adam and to KLAS for giving me the opportunity. Adam encouraged me to get your ideas and feedback and send them his way, which I'm happy to do as well.

Give me some history of KLAS, including how the company’s name came about.

We just celebrated our ten-year anniversary in October. Our principal founder, Kent Gale, was working for a software vendor that was about to go public. Their discussion was how to make their numbers look better and how to cut their support staff to make those numbers.

Having been a salesperson who had seen the benefit of great support, Kent wondered why no one measured the level of support vendors provide, catching them if they dropped it or patting them on the back if they improved. A couple of years later, we had data to share and it was off and running.

KLAS had four founders. Kent Gale, Leonard Black, Adam Gale, and Scott Holbrook. I’d say our crack marketing team came up with the name, which is made up of the first letter of each of their first names, but there was no crack marketing team [laughs].

Several people had tried something similar, but they were always one-shot deals instead of ongoing. Our contacts with providers can’t be hit and run. They need to be positive enough to have them want to participate again next year.

We have 62 employees as of the last count. We originally thought it would be maybe be five full-time people doing research. Every step we take, we get pushed into more nooks and crannies and corners. We love it. We want to be able to ask the questions asked of us. We thrive on finding the truth. That’s when you see the high-fives between offices, when we find something that was kind of a secret in the industry, whether it’s positive or negative.

Who actually owns the company?

Those four original owners, plus there are three or four additional that have gotten ownership over time, employees here. Also Ralph Reyes, who now works for TheraDoc as of about three months ago.

What steps are taken to ensure that KLAS reports are accurate and fair?

That’s a tough question to answer in a sentence or two. We have lots of safeguards in place.

The first is that we always have to take the data as we get it. Whatever the CIO or respondent reports back, that’s what we report to the industry. We don’t clean it up a whole lot. Some of the commentary is fairly off the cuff.

We consider ourselves a research company and aggregator of opinions. People ask us all the time what our opinion is. It’s one of our written commandments: KLAS has no opinion. Only the data speaks.

We have a methodology we follow in every interview. What does the vendor do especially well, what could they improve on, and what does the future hold with this vendor? Based on those answers, we’ll go to other questions to drill down. Or, if the respondent answers questions in a strange way, like all positive answers except in one area, we’ll ask them specifically what the challenges were in that area.

One of your readers said they were challenged by KLAS when they scored a vendor highly, saying they were told that others weren’t scoring them that highly. We never compare one person’s answers with another’s. We’d never say, “Others don’t score it that way, so why did you?”

Anyone who scores off the chart either way, like all 9s or 1s, we may drill down a little bit more to find out what’s really happening. Most vendors aren’t either horrific or perfect. We push on surveys that are extreme just to make sure the person knows why they answered that way. If they confirm that it’s really their experience, we capture that and move on.

KLAS has come down on vendors in the past who intentionally misused their survey results. How common is that and what problems have you had?

It’s no surprise that some marketing teams like to push the limits on what they can do with KLAS information. We’ve put a process in place to publish clarifications any time a vendor has used data that doesn’t seem to be as clear as it should be. It’s a challenging process because so much data is used out there, maybe in a PowerPoint or something for internal consumption, and we even try to clarify those when necessary.

Most vendors today are very good and aware of our data usage policy. They run by us almost everything they’re going to publish externally. Sometimes new vendors have to get a clarification or two that we’re serious about how they use the data.

Some people have suggested that we should stop allowing vendors to use the data at all. We really want to pat those vendors on the back that have done a great job and let people know when they’ve had challenges. It’s a fine line to walk.

Anyone who participates, by rating one of the products we cover, gets access to the unfiltered data and can see the rankings and compare vendors. We want them to be able to see the real information without having it filtered by whatever vendor is presenting it. We probably have over 20,000 people, maybe significantly more than that, that have access into our database, providers who can check their own data.

What are some marketplace and customer trends you’re seeing?

We can see how many times each market segment is looked at by providers. The most downloaded over the past year were ambulatory EMRs, with changes in the Stark law and other government initiatives and the maturing of the products. Not surprisingly, it’s getting more play. That’s #1 far and away at what people look at on our website.

A less obvious one is surgery management. A lot of people are looking at upgrading old surgery systems. For many vendors, even McKesson and Cerner, that’s their most downloaded product, even with broad offerings. That was unexpected for us.

KLAS wasn’t all that strong in ambulatory EMRs. Did you have to beef up that area?

We have expanded hugely in the last year or two. We put out a study in the one to five physician space that we haven’t historically covered, vendors like eClinicalWorks, Greenway, and e-MDs, not the enterprise-type vendors. We now have probably 20 to 25 vendors that we rate in the ambulatory world that we wouldn’t have rated three years ago.

We’ve been pushed deeper into the bowels of the ambulatory world because of the small practices looking for solutions. We plan to break out and dig deeper into the ambulatory world, maybe even reporting by specialty someday. If you look at how much data that would take, it’s astronomical, but we’re getting pushed by a lot of organizations. We’re getting lots of questions after the Stark changes, even though that doesn’t mean there’s all kinds of dollars to spend.

Does KLAS want to be the Consumer Reports of the industry?

I don’t know if we’ve ever thought that we want to be that. We get called that a lot by providers, but we don’t call ourselves that. It does conjure up in people’s minds a specific unbiased rating system.

One thing we don’t plan to do is to rate or test the products ourselves like Consumer Reports does. There are huge obstacles to getting and testing the system in some way that would be meaningful. We do plan to be more and more comprehensive. We’ve been doing medical equipment last year, like 64-slice CT and MR. A lot of equipment that has sophisticated software on it is on our radar today.

KLAS measures perception. Is that always accurate and relevant to a prospective customer?

We’ve found that the information people have is surprising accurate, especially when they use multiple vendors and products. People with experience with two or three vendors have a clear idea of which vendors they would call the best or who they have challenges with.

The muddy middle is a challenge, but it’s interesting to see the vendors who are at the top, who will do anything to solve a problem for you, and the vendors on the other end who aren’t interested in helping you.

Sometimes we get a call from providers about vendors on their short list and they find that one of their choices was #9 or 10. Or, if their vendor is at the top, they can go to the board and tell them it’s highly rated by peers. It’s almost impossible to articulate what the difference between close scores will mean. That’s why we collect the commentary.

Most of the scores are fairly low on the nine-point scale, so that a 6 or 7 is good. What do you think that means?

Take a vendor like McKesson. If you take all their information together and compare all of Siemens and all of GE, almost everything would go right to the middle if you encompassed all their products. What you see as you break it out is that McKesson has exceptional products, like their PACS solution they acquired from A.L.I., which has incredible traction and performs incredibly well. Other products from McKesson don’t score as well. In fact, even Paragon, which scores Best in KLAS, was the lowest rated product in our whole study five years ago. They’ve clearly taken an effort to bring it up.

In the muddy middle, you can find highs and lows that tell you something. The problem is that, for providers who buy from single-source vendors, the only one that stands out is Epic. Some vendors are in the honeymoon phase. Epic is closer to the beginning, but also doesn’t seem to have had a lot of drop-off even with a large expansion in their customer base. Every Epic customer I talk to, I drill down and ask if there’s any drop-off in service since they signed on. They’ve kept the high level up so far.

Products seem to swing wildly from one report to the next, with some products going from first to worst over a short period. KLAS says that’s due to their changing performance. Cynics say it’s because the rankings were statistically questionable to start with.

I don’t think we can always say that with absolute confidence. For the most part, vendors don’t move that much. They stay pretty constant.

For McKesson Paragon, they were the worst-to-first story, but it was over five years. It took them a long time and a lot of effort. We’ve talked to every Paragon site in the county. People can talk about small sample sizes, but we’ve talked to every one and we have a high level of confidence.

For some products, we try to indicate by the number of checkmarks in our confidence score how much the next survey could impact the scores. One check of confidence instead of three means there could be more variability in the next survey, especially if it’s high or low score. With three or even two checkmarks, there’s not a lot of change, and if there is, most of the respondents are saying it’s better or worse.

I don’t think we can ever say with confidence that they’re in fourth place and not third or fifth, but hopefully it’s within some reasonable statistical range. I agree that if someone buys #3 instead of #4, that would be a mistake without other significant homework. I don’t think most people do that.

It is frustrating to us when a product is rated #1, people go and buy it, and the vendor has challenges in delivering. We’d like for people to perform well and count on KLAS ratings. Vendors may get acquired, lose key personnel, or stop putting money into the system for whatever reason. You also have sampling error, which we can never get rid of other than by talking to every site in the country.

People accuse the Best Colleges edition of US News and World Report of changing the top schools just to make everyone buy next year’s edition.

We don’t have to make anything up to make this industry change quickly. For example, Stentor gets acquired by Philips and then you have all the challenges of integration. Any time a vendor gets acquired, it’s tough to keep up the level of service. Philips has done a reasonable job, but it’s challenged. It never crossed my mind to change something just to have something to sell.

Top-ranked vendors who are little known get acquired in many cases, either because they really are the best or maybe just because it looks good to buy what someone says is the best.

It’s no surprise that highly ranked best-of-breed vendors are acquisition targets. Who wouldn’t want to acquire the best?

It’s not our goal to generate acquisition interest. It is a goal to let people know about products that do well. For example, Unibased Systems Architecture is one I’d never heard of until we heard it enough times and did some digging.

We did the same thing with Epic when it was rated #1 in our second report. Kent called Judy directly and told her we were about to publish something, saying, “You’re number one and we don’t even know who you are.” We flew to Madison to check them out. We go through the same process to check these vendors out and ask the appropriate questions.

Would the company be willing to have the survey and ranking processes audited by an outside expert for statistical validity?

I would say absolutely. In fact, you asked about the origins of KLAS. Leonard Black is head of entrepreneurship at the University of Utah. In a previous life, he was a professor of statistics at the Air Force Academy. We feel we have a strong statistical base.

We have a 12-person advisory board that’s the first people we go to with questions on how we do things. They’re providers from across the country. One of their suggestions at the last meeting was that we do an independent audit, trying to figure out how that would work and how to make it real and meaningful.

We’d absolutely be willing to have an audit on how we get to the scores. We want to be as transparent as possible in putting the providers’ input into meaningful format and rankings. If you get criticism, please send it to me so we can improve. My job as COO is to make sure we continue to get better and better at this.

Organizations that offer services similar to those of KLAS don’t usually get paid by the vendors they evaluate. Is that a conflict of interest?

I would say we have one of the world’s strangest business models, where internally, if you ask anyone at KLAS who our customer is, they’d tell you it’s the provider. That sometimes irks the vendors because they pay a reasonable amount of money to have access to the subscriber data.

One vendor, as a mistake, sent us an e-mail intended to be internal that said, “Doesn’t KLAS understand who the customer is based on how much money we spend?” We hold that up and cheer. The vendor is not our key customer. The provider is. We frame every vendor question in terms of, “Will it help providers make a better decision?”

Vendors throw out ideas that would make them look the best. Just about every vendor subscribes to KLAS. I’ve heard folks say, “Clearly the vendors that pay you the most are going to get rated the best.” You know which vendors which pay us the most and it won’t be USA or DR Systems. It will be a GE or McKesson or Siemens. Although they all have some good products, we’re not afraid to say someone’s #1 as small vendor from Chesterfield, Missouri. Anyone using USA pretty much calls us and tells us, “You’re right on.” They’re the biggest unknown that goes above and beyond.

How much of KLAS’s revenue comes from vendors?

Not all of it, but over half is from vendors and also consultants. Almost all the consulting groups subscribe, investment banking groups, and most of the large hospitals. We have a varied base of customers, but the vendor portion is a big piece because we count on providers more to provide their input than giving us money, although many do subscribe when they’re making a buying decision.

At this point Adam asks what I think of KLAS since I was formerly a subscriber and participant. I said that I never questioned the company’s relationship with vendors, but I did think that a minimal amount of scientifically validated information was massaged endlessly into meaningless graphs and charts to make it seem more useful than it really is.

Maybe it’s too easy to drill down to a gnat’s eyebrow when there’s not enough data. For CPOE, for example, there is. We do a 100% sample and ask deep questions.

You’re right. Sometimes when things are close, we try to find out what is the differentiator. If you look at raw scores, not much comes out, but you might find out they’re strong on the west coast because that’s where their support center is. Clearly it’s a fine line of when you stop or go one more iteration to find the golden nugget. We’re open to input or suggestion, especially since we see providers as the customer. If we can make it easier, that’s what we want to know and do.

What do you think about HIMSS Analytics?

We don’t see them as a competitor at all. I consider Mike Davis as a good friend of mine. We have a partnership where we share data with them. They’re really measuring who’s using what, not really anything on how happy they are.

We think that it’s great to have a census out there in who’s using what since we’re providing the satisfaction meter. Some of those tools would go well together, the HIMSS Analytics market share and our performance reviews. Dave and Mike have done a lot to clean up data that’s questionable. We’ve called them on data that we didn’t find mapped to the industry and they’ve cleaned it up. It takes a lot of effort to get what they collect. If I’m not mistaken, they don’t have any performance information.

Vendors seem to be changing their operation solely to improve their KLAS scores. Is that the best use of their resources?

I was very impressed with Ed Daihl’s answer to your interview question about SIS’s KLAS scores. It was a great answer, not defensive, just “We realize we have issues and we’re going to fix those.” Not every vendor will take the first step to admit they have a problem to solve.

A lot of vendors could do a better job in talking to us about what the big issues are. They may look at our scores and see a low score on quality of documentation, so they put money in that, but maybe that’s not someone’s real pain point.

KLAS could do a better job about sharing what could improve scores. If you’re frustrated, everything is bad, response time and support callbacks, even though response time is really OK. There’s a halo effect on how well the vendor is doing. I think the vendors could do a better job drilling down into which ones are most important to providers and we as a company should do a better job to help them.

Our job is to help them make providers happy and if they’re not doing that, we’re missing the target. KLAS has to do things that make a difference for providers, even when it’s challenging, like interoperability, which isn’t easily solvable in minutes.

What can we expect from KLAS in the next few years?

I would say providers should expect deeper information on the areas they really care about. We’re about to publish in the next two months an independent nursing study on usage of nursing products that nothing would even compare to. We’re trying to cover areas that haven’t been looked at much, like what do chief nursing officers think of their core clinical products.

You’ll see more on the equipment side, like smart pumps, patient monitors, and potentially lab instrumentation. IT is getting to own a lot more of this equipment than they have in the past, like imaging equipment that connects to the network. We’re going to get into anything the CIO touches and maybe some things they don’t that maybe a cardiology director or radiology director would care about. That includes service offerings, like partial outsourcing or outsourced coding or transcription. You’ll see a lot more depth and breadth from KLAS.

We don’t want to hide from criticism. We want to fix it. Blogs are unique. Anyone can pose as anyone and say anything they want. There’s not a lot a vendor can do about that, except to do their best and keep smiling.

I’ve told you this before. I like reading your stuff, especially the HISsies. I print that out and hand it to a lot of people. It doesn’t make me always agree, but often we find your comments very interesting and if nothing else, thought-provoking about what something means to the industry.