Spectacular interview! And I personally envy Justen's ability to present
the facts in a comprehensible order and refer to the supporting documents
on demand. These skills can be of enormous value to any organization, but
they're incompatible with groupthink. Kaiser is being bogged down by
groupthink, and Kaiser's leadership has been remiss in permitting the
enforcement of groupthink and the arbitrary punishment of dissent for so
long. The next time anyone hears "he/she doesn't understand the way things
are done at Kaiser", they will shiver down to their very bones.
Like every 25 year-old, Justen seems to have a lot of strong opinions. For
example. he states "The old system was reliable, even though the IT people
hated it." Since he has only worked at Kaiser for two years, what does he
know about the old system? In the Computer World article, he is quoted as
saying: "Epic simply cannot scale to meet the size and needs of Kaiser
Permanente." How does he know that? Is he an expert in systems analysis? In
industrial engineering? In thin client deployment? Has he done any
becnhmarks or measurements of the Epic software? Or is all of his certainty
based on the incomplete reports done on the Epic software two years before
the install at Kaiser began? Does he believe that the software Kaiser
received didn't evolve a bit from the time the reports were written?
Dear Mr. HIStalk: With all the Epic issues chronicled in the blog, I feel
bad being cynical but is it just coincidence that SCI Solutions became a
Gold Sponsor at a time when a major Epic client like KP is announcing
"scalability issues?" Wheren't SCI's "smokin" KLAS scores topped by Epic's
equivalent product? Pretty tough to crack a big high profile Epic client
unless Epic slips on the proverbial "banana peel." This appears like those
lawyers that chase ambulances. Please say it ain't so Mr. HIStalk!
"is it just coincidence that SCI Solutions became a Gold Sponsor at a time
when a major Epic client like KP is announcing "scalability issues?"
Wheren't SCI's "smokin" KLAS scores topped by Epic's equivalent product?"
KLAS Reader you are cynical but also wrong. We at SCI have been in the
process of advertising on HISTalk for some time now. As will happen it
just took us a while to get everything ready and the timing with any EPIC
news was truly a coincidence. SCI and EPIC don’t compete for any sales (we
wish we did). Thanks for thinking of us in the same context as EPIC which
I will consider one of the great success stories in HIT.
Some things from an in-hospital nurse:
//So why is Justen so sure about facts and events from before he was at
Kaiser?//
"By October, we were at 60,000 user hours of downtime and early in November
we were already at 35,000 hours. It's a combination of poor network
infrastructure and poor Citrix infrastructure. To say it’s all due to power
outages reflects a poor understanding of what’s going on."
35,000 hours out of how many? 60,000 hours in October out of how many?
And the denominator is (drum roll) about 8,500,000 user hours / month.
So based upon Mr. HIStalk's comment about wanting to learn from the KP
situation, does the majority of HIStalk readers also want Epic to
ultimately succeed at KP and other health systems where it is being
implemented or live? Or, is bad press and implementation struggles too
attractive to resist our attention and discussion? My opinion is that it
would be good for all of us for Epic to consistently succeed at a number of
installations. Looks like the guy from SCI is firmly in the Epic
camp....how about the rest of you out there?
This Justen guy is CLUELESS technically. KP is getting 70+ EPIC users per
Citrix server, and there are many implementations with tens of thousands of
users that are not KP. (Cingular) This is a dumb kid with a few facts and
a ton of rhetoric, misinformation, and uneducated opinion.
Ok. Someone is seriously trying to mislead here. The uptime requirement
for a system like this is 99.99%. The fact that Halovorson's response
suggested the goal was 99.5% was the point at which any knowledgeable
person knew that he was spouting BS. No mission-critical system has uptime
goals so low. Customer service systems have goals higher than that
(typically on the order of 99.7-99.9). Phone systems and credit card
systems are in the 99.99 - 99.999 range so don't spout spin about the
denominator when these numbers are an absolute embarassment when you're
talking about a healthcare system. No wonder Justen was frustrated when
this blindingly obvious fact about IT deployments is ignored (and defended
by some anonymous KP stooge in the comments).
To the poster who suggested 8.5 million user hours per month... It looks
like you either have no idea what you're talking about, or you've based it
on the number of Kaiser members. In either case, you're wrong. KP has
somewhere south of 3.2 million user hours per month at this point. At the
end of October, KP was seeing something like 98% uptime, based on user
impact hours. I read that the first week of November was already at 35,000
user hours of downtime... That would put them on track for 95% uptime.
Not acceptabled for a hospital, as Datacomm Guy pointed out.
If we use 8,500,000 potential user hours as the denominator, then
availability is around the area of 99.0%. I wouldn't want to be the patient
affected by that 1% or 1-out-of-100 downtime. And this is way below
Kaiser's goal of 99.7% (which, as someone pointed out, is a low goal in
itself).
Datacomm guy has a clear point. As of now (from the data that I've seen
which is the same as ComputerWorld has) HealthConnect is running in the
98-99% range. It is nowhere near the "five 9s" desired uptime other than in
Colorado, and in some cases below 98%. So is that good enough? I tend to
doubt it too.
Am I the only one concerned about how favorable these contracts are for the
vendors? Epic has made a MINT--not just from Kaiser, but because of
kaiser. Yet Kaiser stands to lose their shirt on this. Wasn't the
original announcement a $1.8 billion deal (after writing off around $400
million)? Now it's $3 billion? And it's up (and unreliable) in 2
hospitals... What the f--- is happening? Is Kaiser already hemorrhaging
because of this, or did I miss some other press release. If not.... More
than a billion dollars over budget should at least raise a few eyebrows.
Will someone do some fact checking for crying out loud? Justen Deal is a
Publication Project Supervisor in the Health Education and Training
Department, not a Project Manager, and definitely not directly involved in
the implementation of such a system. Does he even stop to consider that
once running KP's HIT system will most likely be one of if not the largest
private network on the planet? Its amazing to me to see so many people eat
up his story so readily without cold, hard, facts! Most of what he says is
pure speculation, such as "Epic will not scale" in his e-mail. How would
anyone except some of the top systems analysts Kaiser has know anything
about the issues surrounding the implementation of a system meant to hold
so much information and be accessed by so many people in such a large
geographical area? Why is no one asking questions!?!?
OK, with all these uptime statistics being thrown around, can someone
please tell me a good number for the implementation phase of a major IT
project like this? 99.99% may be a reasonable standard for a system that
has been in place for a while, but what about a system that is being rolled
out and the bugs are being worked out? Is 98% reasonable then, or not?
Attacking Justen for not being part of project implementation is just
misdirection. The flaws here are so OBVIOUS that anybody can safely
comment on this as a clusterf***. To the person who asked what is
reasonable, it works this way in every deployment I have been involved
with. You test the hell out of it in a systems lab environment with a test
plan covering every possible scenario including having people pound on it
with the intention of causing it to fail. Bug reports are created and
classified by priority. No priority one bugs can be present for you to
move to the next stage. Only priority two bugs with known causes and fixes
on their way can be extant. Priority three bugs are fuzzier and are based
on a subjective feel of the number and quality as to whether you can move
forward with them. After system testing, you do load testing and it better
meet the goal since this is a controlled environment. The load testing
includes all elements including routers, servers, power, etc. Don't give
me any BS about those things not being included in the uptime goal. At
this point you go live at a beta site preferably in a dual mode although
for this project I would guess that would be unlikely. No way you move
forward from beta to controlled introduction (2 - 20 sites) unless you are
still meeting every single one of your quality criteria. All this time you
are collecting, collating, and learning. I can't tell you exactly, but
each phase is measured in months so the load testing may be a real or
simulated 3-6 months of use. Beta certainly is 2-3 months, if not longer;
CI, the same thing. Finally, after you are convinced that this all works
the way you want it to you go live. In the meantime, you are doing other
stuff like focus groups with the users and maintenance to fix some of those
priority 3 bugs that you let slide before beta. You don't have to be any
kind of expert to condemn this implementation based on the published
acknowledged FACTS so don't jump on Justen for not being an expert. It
just shows you for a mealy-mouthed apologist, hypocrite, or idiot (you
choose).
Sorry, my comment should have been more succinct. No, 98% is not
acceptable at any stage. You structure your roll out so you meet your
uptime goals at every stage. The hope is that the numbers go UP (from your
base requirement, in this case, 99.99%) as you increase the denominator and
you gain experience (and you users gain experience). That's how I knew the
commentor spouting about the denominator was full of it.
Epic's a good system as pointed out by folks who have actually worked with
the system in other organizations and by KLAS reports. The scale of Epic at
KP is new ground for Epic (or for any other vendor out there today is my
guess) but they've been innovative enough to get around limitations with
hardware and other technologies. When you have many moving pieces, all it
takes is one part to fail to make it appear like the whole system is
failing. I agree that 95% is too low of a uptime to be happy with and even
99.7% might be low. So many people are gloating at the failures (and
perhaps perceived failures) of KP+Epic but BOB LABLA makes a good point
that we should all be rooting for KP+Epic to succeed because it would be a
good for everyone. Maybe Justen's motives were sound in wanting KP to have
a good system but I think he's gone about it the wrong way. KP staff
reading all this bad press about their organization must feel disheartened
that all their efforts and successes are not being recognized because there
are successes based on what I'm reading. Whether it's one hospital or all
hospitals, every rollout, every user starting to use a EMR, every
automation of a paper process is a success story in itself. I'm sure that
prior to this fiasco and long after this has died down, the staff and
management of KP have been and will continue to work hard to provide a good
system and sound infrastructure for their organization and users.
It is known within Kaiser that Garcia has initiated an HP-like information
gathering program to identify employees releasing information to blogs.
Noone wants a successful implementation more than I. I am hospital nurse's
husband and trying to explain the benefits of technology gets old pretty
fast when she points to her grief at work from cra**y implementations.
Let's not pretend, Kaiser's current implementation sucks. You don't have
to be an EXPERT to see that so the person attacking Justen for not having
credentials is as full of it as Halvorson. As Justen makes clear in his
interview, it's not EPIC's or KP-IT's fault or even Citrix (although I'm
pretty biased against Citrix from a basic architectural philosophy). It's
a failure of leadership and management to the extent that they have lost
sight of what they are trying to accomplish and have allowed a really lousy
program to move forward by sheer inertia since noone apparently wants to
point out the emperor has no clothes. Justen has done that and we can see
the KP trolls and apologists on this site alone trying to argue otherwise.
99.5% uptime? Get real - that sucks. Billions over budget? Get real -
that sucks. 100% Citrix implementation with no fat clients? Get real -
that sucks. Money of the magnitude that KP is spending along with
executive buy-in at that level should not result in faulure and does not
have to even at this late date, but people have to be willing to
acknowledge problems instead of bulldozing forward in the mistaken belief
that acknowledging problems invalidates the program. I don't know enough
about EPIC to agree with Justen about dumping it; I suspect that would be
the wrong approach and that a collaborative solution with EPIC would be the
right approach. On the other hand, I totally support him in publicizing a
really awful situation that Kaiser was not willing to address because they
believed it to be politically infeasible. He probably even got it right
that the political raodblock was Halvorson who responded to his concerns
with a BS broadcast e-mail.
Did anyone else notice in the ComputerWorld article that the KP-IT Quality
dude is named Neidermeier? There is your problem right there.
Datacomm Guy: The problem with Justen is not that he is pointing out
problems at KP, the problem is that he is using the wrong information,
opinions stated as "facts", and the wrong aproach. If you read his letters
at fixkp.org, he is basically asking for the head of Halvorson for
"forcing" Epic on KP, not for determining the reasons for the perceived
problems, or measures to improve the situation. His letters to the board
were for the investigation of how Epic was selected, and he got a response
on that.
Has the KaiserTurf begun?
Scaleability is complex. When someone says "it will not scale", this is
actually rarely the case. The truth is that almost anything will scale if
you throw enough hardware at it... but every extra box brings problems. I
call Citrix the "necessary technology evil" because it enables many users
to overcome software architectural limitations. Cerner implementations
often use it too. The truth is complex here. Justin himself says that
Epic is "perhaps the leader in the HIT sector". Who has actually proven
scale out to so many users- 100k to 180k, depending on who you read, in
this sector?
At the end of the day, Epic is architected for many things such as
responsiveness and user experience richness. I wouldn't have put scale at
the top of their list, but then they're not entirely responsible for
implementation architecture and running of the hardware.
On KaiserTurf, hopefully Garcia has learned from Hewlett-Packard.
Unfortunately, I suspect that Halvorson and Garcia are looking out for each
other - which shows the inherent problem of a Board member also serving as
the Chief Compliance Officer and involved in hiring the CEO.
KP may be in a better situation if they had an effective Ethics program to
support an open culture. I suspect Garcia's ethics officer lacks the will
and clout to take on senior leadership and ensure the right things get done
- such as ensuring there is no retaliation for reporting problems.
Just by comparison, HP might have avoided much of the pretexting scandal if
its chief ethics officer, Kevin Hunsaker, had asked more questions and
demanded more answers instead of kotowing to his boss (the general counsel
who resigned) and the HP Board. See:
http://www.forbes.com/2006/10/23/leadership-ethics-hp-lead-govern-cx_hc_102
3ethics.html
Doesn't Kaiser have a bloated ethics and compliance
department? Last I heard there were upwards of over 60 people in the
department and that's not counting indirect reports and related network
structures. That's huge! And being headed by a member of the Board they
still can't instill a better corporate culture?! What have they been doing?
I think Justen is calling it as he's seen it, and I won't hold his youth
against his objectivity in the matter.
KP-IT and planning seem to be polar opposites, how many contractors had to
be let go early March due to a "slight" overage on the budget? Some had
only been brought on weeks earlier, and to the tune of how many tens of
millions? (talk was between 70-90) KP-CIS was the exact reason I choose to
never work healthcare again, best to stay the course in the banking sector.
Now that everyone is talking about uptime percentages, what is an
appropriate percentage as a system is being implemented? Because that's the
issue here. It's one thing to expect 99.99% uptime for an established
system in which bugs have been worked out, it's another to expect it as a
system is being rolled out. So what is the standard there?
I think lost in all the furor is the fact that HealthConnect is working
beautifully in the Kaiser Permanente regions where it has been introduced.
Today, members in Colorado can e-mail their doctor, find their lab results
online, check the doctors' notes from past visits, and see a list of their
allergies. Similar features are available in Hawaii, Oregon, Ohio, Georgia,
and the mid-Atlantic states. For several years in Colorado, patient records
have been paperless, which has meant fewer errors caused by illegible
handwriting, easier hand-offs to specialists, and just-in-time information
in the emergency room. Colorado pioneered the CIS system that was later
scrapped in favor of HealthConnect. I don't think you'll find many doctors
here wishing they still had the old OS2-based CIS system. Outages happen,
and we hope they will happen less frequently. But if the medical world
waits until system outages are extinct before building electronic medical
records, we'll remain in the 19th Century forever.
Justen obviously knows very little about IT and less about Citrix software.
He apparently doesn't even know much about what Citrix is doing on site
with Kaiser. This money quote is like a billboard for Justen's lack of
knowledge. "We're using (Citrix) in a way that's quite different from the
way most organizations are using it. A lot of users use it to allow remote
users to connect to the network. But we actually use it from inside the
network." This sounds like something that could have been said in 1998. It
is certainly not true today. Justen should stick to working in Healthcare
Education and leave Information Technology Project to people who actually
know what they are doing.
Here we use Epic and Citrix for 99% access to our Epic EMR system in a
large environment. And can say for sure it works quite well.
Justen has his facts straight. He was very straight forward in letting us
know he was not on the impementation team nor is he IT. Let's not attack
him there. I live in a different region and was on the implemetation team.
It was a fight every during build, validation and design to make KP
mangement understand that Citrix was doing a job it was designed to do.
The team repeately vetoed IT and numerous other presenters, to include the
Kaiser Coalition of Unions, when we asked for changes. By no means should
Citrix be to blame for KP's problems, unless you add every other vendor
involed. The blame is with poorly written contracts with vendors, lack of
communications within the KP infra-structure, micro-management of projects
that lead to information loss and the large amount of siloed information in
a build that should certainly remain transparent. The finger should be
pointed at the management that ignored all that opposed it, choose to cut
corners when thorough testing was necessary and spent KP money at least
once a day to feed the group.
I work for KP, and there's no way HealthConnect is up 99% of the time! The
server is always busy. All of this denial is putting patients at risk.
Truth be told:
http://www.brianmadden.com/content/content.asp?id=639
I work at Kaiser Sunnyside Hospital and I love Epic. It is the best for
doctor,nurse and patient.
I recently left Kaiser after over 5 years in KP-IT. I left because the
management chain all the way to Dodd could not effectively manage
themselves out of a paper bag.
The worst thing about all this? Only the people in charge have all the
data from all sides of the discussions...and they're not sharing/asking for
help.
While I agree with the assessment of KP-IT, I don't think the issue of IT
project management on the business/medical group side should be overlooked.
That's where a lot of the bungling is going on. At Kaiser, the main route
to career advancement for physicians is management/administration,
especially IT management. There are too many physicians with a
vested-interest crowing for "physician oversight" in order to create
leadership positions for themselves and not enough people thinking about
whether these people have the necessary experience/qualifications to lead
IT projects. In my experience, a lot of physicians *think* they know more
about IT than they really do. They should stick to UI testing - and even
then keep in mind that their judgment might be tainted by lack of
experience.
As a Kaiser Permanente technical staff member, I am very familiar with the
extreme technical deficiencies prevalent within the organization. I am
also sorry to say I too have first hand knowledge of the draconian tactics
employed by certain egomaniacal leaders and their selected minions.
I can clearly remember the shock that went through our area when we learned
about the seemingly abrupt Epic decision. As we were learned more about the
system we became worried. The early information talked up Epic's success in
small med centers. According the the Epic head, all it had to do was "scale
up". Yeah, right. Another of her points was that few changes were expected.
Yeah, right again (if you know anything about Kaiser). It was worrying when
we learned that we were targeting roll out of the replacement for the
application I was working on at the time to start in six months and that
upper management wasn't interested in discussion. As my manager reported
from a meeting with her peers with their manager, the word from above was
"Believe or Leave". A lot left.