Subscribe to Updates

E-mail:
Name:

No title

Search HIStalk

 
WWW HIStalk
No title

Blog Status

  • 6 yrs 33 wks 4 days old
  • Updated: 8 Dec 2009
  • 915 entries
  • 2,025 comments

x
Platinum Sponsors










x
Gold Sponsors







HIStalk Quotes

An Exclusive Interview with Justen Deal of Kaiser Permanente

posted 11/13/2006
HIStalk
Justen Deal is every bit as thoughtful and articulate as he seems in his interviews and e-mails. This isn't some out-for-revenge nutjob employee or would-be celebrity craving his 15 minutes of fame, as far as I can tell. Lots of folks are trying to poke holes in his story of an EMR crisis at Kaiser that he e-mailed to employees (only after the board ignored it), but I haven't seen anyone yet prove any of his facts to be false. Every time someone says, "he's wrong, everything is peachy" he calmly cites another internal document that's available from one source or another to confirm his statement.

Maybe I enjoyed talking to him because each of us probably wistfully remembers what it was like to be 25, not yet conditioned to take the paycheck and keep your idealism to yourself, when wrongs (perceived or real) should be denounced by all. I'm fearful that he won't get his wish to go back to work at his beloved Kaiser anytime soon, but I'm sure he has a bright future nonetheless.


Are you surprised by the attention you’re receiving?

I was expecting a fair amount of internal communication. Someone said I was naïve in thinking the e-mail wouldn't go outside. Most of what we do inside of KP stays there. Although I guess I’d seen some of the HIT blogs in the past, I hadn’t recognized how large and vocal the community is.

The mainstream media has been fairly decent in its coverage so far, but I was discouraged to read that some of the quotes from our PR folks said HealthConnect was working well. Our PR folks have attempted to spin the figures and say that everyone’s happy with the system. Up until the Computerworld article this morning, it was just what I’d said, or what others who could be demeaned as disgruntled amployees said.

The Computerworld article brought up the 722-page report of all the outages from this year. It points out the severity, length, and frequency, and indicates in our own words what the impact has been on member care. The report makes it tough to keep saying the same thing.

The outage report seems to be a bit of a smoking gun, whereby Kaiser says in its own documents that patient care has been impacted by system problems.

Frankly, I think we’re very fortunate that the system has not been rolled out as pervasively as you might think. Only two hospitals truly depend on it for what we would typically consider to be mission-critical patient care. By and large, they’ve perfected their downtime procedures.

Any benefits we would have seen have been eliminated because they ignore HealthConnect and use paper just as much. Even though we’re saying we’re paperless, paper is perhaps even more important than ever. Virtually anything that's put in is printed out right away, because there’s no reliance on the system being up ten minutes or an hour from now. The amount of lost productivity is monumental.

What’s your status with Kaiser as an employee?

I’m still on paid administrative leave. My understanding is that they had looked to see if there was any sort of policy broken by sending out the mass e-mail. They weren't able to find a policy that conclusively prohibits mass e-mails. As far as I can tell, they’re just waiting or looking for anything else that might make it more feasible for my job to be eliminated.

Did you send your e-mail to the whole company or just part?

Kaiser has four separate internal e-mail systems. A third of users have Outlook/Exchange, about half use Lotus Notes/Domino, and others have GroupWise and AIO. Any sort of locks on Lotus Notes wouldn't have had an impact. Essentially, I individually addressed an e-mail to 50,000 people or more. The spread to basically everyone else in the organization was by those folks forwarding it to their colleagues.

Were you sorry to see Kaiser CIO Cliff Dodd leave right after your e-mail went public?

He didn’t make the decision to tie the success of HealthConnect to a single vendor. He’d worked fairly hard during the past few years to make the system work. It wasn’t happening, but it wasn't his decision. If there was anyone who knew the ins and outs of our technology, it was him. To see him leave so quickly and without any goodbye and certainly without a lengthy period in which we could rely on his knowledge was unfortunate.

What motivation do you suppose George Halvorson had in bringing in Epic?

Dr. Wiesenthal, in Matthew Holt's interview, said it was a unanimous decision. That is absolutely not the case. Many were concerned about scalability and productivity issues. On George Halvorson's first day, he cancelled the KP-CIS project. It was clear that Epic would be the solution from the get-go.

Who wrote the internal reports warning of scalability issues?

One of the reports was an overview of vendor solutions. It was a never-completed draft dated November 5, 2002. The purpose was to compare Cerner and Epic. By then, we had already had lengthy discussions with Epic in terms of dollar figures, etc. We signed the contract first part of December. The report wasn’t even completed by then. That's kind of interesting.

Some of the key findings of the report was Epic's untested scalability, their minimal experience with mega-implementations, and whether or not their systems were HIPAA compliant. The scalability and HIPAA issues are why we put Citrix as a middle ground between the two.

It was also noted that each individual linked system instance would be two or three times as large as the vendor's largest deployment. It said we would have to prototype it. The question is, did we ever even complete the linking of the distributed systems? The recommendation was that KP take an active role in the distributed architecture design and that both vendors' contracts should include risk-sharing for development milestones.

Another document was dated January 20, 2003, just before we publicly announced the Epic decision. It was also in draft, never completed, by Brian Birch and a team from the CTO's office. By then, we had already figured out that we were going with Epic, but thought that parts of KP-CIS might still be around. The recommendation was that Citrix should be used by no more than 20% of end users connecting. We're now at 100%. It's the only way to connect.

The justification for Citrix for the 20% was that Epic could overcome their security and scalability issues and that wasn't the case. Citrix was tbe rolled out only to avoid delays with putting workstation management tools on the desktop. The report pointed out that, even with 20% Citrix, it would be expensive to build an interim infrastructure and would require more resources. Our engineers never recommended using Citrix as a permananent infrastructure and I don't think they would have even thought it was possible.

The report said there would be a minimal risk with 80% fat client, 20% Citrix mix. It pointed out that the full rollout would be the biggest Citrix deployment in the world, up to hundreds of servers.

How many Citrix servers are being used now?

Thousands. We can only get 12 sessions per server, so that's several thousand.

We certainly went against our own engineering reports. Physicians liked Epic's feature set, but were concerned about scale. Their biggest implementation even today is 1/10 the size of Kaiser. We needed the billing module, but engineers said it wouldn't scale. George Halvorson even said in his e-mail that we needed to let Epic prove itself. That's not a prudent business decision and certainly the engineering reports have proved that. Kaiser's engineers are working night and day to support a system they said wouldn’t work.

Some have said that if Kaiser is having reliability and scalability problems with Epic, it is probably because of how they installed it. Do you agree?

Epic has amazing feaures. No one would disagree. It’s perhaps the leader in the HIT sector. I think the issue absolutely is their scalabilty. It’s a combination of Epic, Citrix, and KP-IT. Laying the blame at any one’s feet is impossible. All are responsible. I don’t think Epic recommended or guaranteed our architecture. I don’t think we have a service level agreement with them, or if we do, it’s not being met.

In today's Computerworld article, Citrix seems to blame Kaiser’s architecture design for the problems there.

Neither Epic or Citrix have taken responsibility. SLAs are nonexistent because of scalability. Neither would guarantee it would work, except maybe in their sales presentations. When it comes to an SLA, neither would put either blood or money on the line and rightly so. You’d have to be a fool to.

Kaiser blames a lot of downtime on power problems. 

We had extensive power outages in the Corona data center. When we began rolling out Citrix to the point we needed to, the 72 clients per server that Tanning Technologies had suggested was unrealistic. We had to ramp up bringing Citrix servers online. We realized that we didn’t have room or power enough in Corona to run them, even enough power coming in from the landline connection or UPS to support the number of servers we already had.

In March, we upgraded the UPS and it went awfully awry. The data center was down for hours over three business days. To say that’s the only issue is wrong. By October, we were at 60,000 user hours of downtime and early in November we were already at 35,000 hours. It's a combination of poor network infrastructure and poor Citrix infrastructure. To say it’s all due to power outages reflects a poor understanding of what’s going on.

Why do you think Cliff Dodd left?

I don’t know. It was abrupt. Even his senior reports didn’t know. The board has known about Tanning since August.

Why is KP-IT not running the HealthConnect project?

I've seen that stated, but it's inaccurate to say. KP-IT is absolutely responsible for supporting the infrastructure. The Corona data center is run by KP-IT. Most of the staff of HealthConnect, the vast majority, are KP-IT employees.

Kaiser isn’t likely to turn back at this point. What did you hope to see happen from your e-mail?

The truth is that it’s not going to work in the way we’re trying to do it. I haven’t seen a single recommendation inside or outside that seems possible. Throwing more resources won’t fix it. Can we get Epic to work? Do we need a summit to try to figure it out? I don’t know. We’ve invested a tremendous amount of resources in it.

We definitely must halt the rollout and remove any critical dependencies on Epic. Moving forward, do we need to look at other vendors, mix other vendors with Epic, or work with Epic to improve scalability?

The old system was reliable, even though the IT people hated it. Can we learn from those infrastructure practices to help Epic scale to meet our needs? I don’t have all the answers. I’ve shared some suggestions within the organization. Continuing to beat the same horse expecting different results just won’t work. This horse is just about dead.

People are commenting how articulate and well-researched your e-mail was, as well as how well you’ve handled the press. Where do you go from here with your career?

I have been with KP for two years now. I really appreciate and care very much for KP as an organization and as an idea. I am happy here and I intend to stay. I enjoy my job. I have a pretty interesting balance between IT and editorial work. I get to use both a Mac and a PC, which I love. I can’t wait to get back.

I believe in KP and HealthConnect as a concept. We have to do it. We have to have an EMR that works. I don’t know that I’ll be a part of that or would want to be, but even with all of this, even with what I would consider to be poorly written and loosely fact-based responses from our executives, I still believe in KP. I know just about everyone else in the organization does too, and rightly so. I hope to be back in my office soon.




1. Gadfly left...
11/13/2006 8:50 pm :: http://corphq.livejournal.com

Spectacular interview! And I personally envy Justen's ability to present the facts in a comprehensible order and refer to the supporting documents on demand. These skills can be of enormous value to any organization, but they're incompatible with groupthink. Kaiser is being bogged down by groupthink, and Kaiser's leadership has been remiss in permitting the enforcement of groupthink and the arbitrary punishment of dissent for so long. The next time anyone hears "he/she doesn't understand the way things are done at Kaiser", they will shiver down to their very bones.


2. Anonymous left...
11/13/2006 11:14 pm

Like every 25 year-old, Justen seems to have a lot of strong opinions. For example. he states "The old system was reliable, even though the IT people hated it." Since he has only worked at Kaiser for two years, what does he know about the old system? In the Computer World article, he is quoted as saying: "Epic simply cannot scale to meet the size and needs of Kaiser Permanente." How does he know that? Is he an expert in systems analysis? In industrial engineering? In thin client deployment? Has he done any becnhmarks or measurements of the Epic software? Or is all of his certainty based on the incomplete reports done on the Epic software two years before the install at Kaiser began? Does he believe that the software Kaiser received didn't evolve a bit from the time the reports were written?

Another quote from the ComputerWorld article: "We're using (Citrix) in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network." I may be wrong, but isn't Citrix the premier Windows thin-client solution? Aren't thousands of organizations using Citrix in thin-client deployments within their internal networks? Does Justen know what he is talking about?

Here is one bit, where he doesn't know what he is talking about: "It was clear that Epic would be the solution from the get-go." That is not true; in fact Cerner were so sure they were already chosen, that not being selected by Kaiser caused them to miss their quarterly estimates, and they hardly ever miss their estimates. If you look at their stock chart, you will notice the huge abrupt dip in Q1 of 2003: http://finance.yahoo.com/q/bc?s=CERN&t=5y&l=on&z=m&q=l&c=

So why is Justen so sure about facts and events from before he was at Kaiser? I don't have any reasons to doubt his sincerity, but could someone, whom Justen trusts, be feeding him some half-truths and misinformation? Are we all being drawn in some internal Kaiser political battle?

I am all for questioning authority, and fighting the system, but you can't have incredulity from ignorance...


3. KLASreader left...
11/14/2006 7:58 am

Dear Mr. HIStalk: With all the Epic issues chronicled in the blog, I feel bad being cynical but is it just coincidence that SCI Solutions became a Gold Sponsor at a time when a major Epic client like KP is announcing "scalability issues?" Wheren't SCI's "smokin" KLAS scores topped by Epic's equivalent product? Pretty tough to crack a big high profile Epic client unless Epic slips on the proverbial "banana peel." This appears like those lawyers that chase ambulances. Please say it ain't so Mr. HIStalk!


4. KlasReaderToo left...
11/14/2006 10:33 am

"is it just coincidence that SCI Solutions became a Gold Sponsor at a time when a major Epic client like KP is announcing "scalability issues?" Wheren't SCI's "smokin" KLAS scores topped by Epic's equivalent product?"

Epic's scores are higher for scheduling. But I don't see your point. SCI does not expect Kaiser inpatient problems to weaken Epic as an outpatient scheduling competitor and decided to suddenly run an ad to steal their business. I want to know about KP's problems because I (we) can learn from them. I don't see that SCI or anybody else sponsoring matters.


5. John Holton left...
11/14/2006 10:51 am

KLAS Reader you are cynical but also wrong. We at SCI have been in the process of advertising on HISTalk for some time now. As will happen it just took us a while to get everything ready and the timing with any EPIC news was truly a coincidence. SCI and EPIC don’t compete for any sales (we wish we did). Thanks for thinking of us in the same context as EPIC which I will consider one of the great success stories in HIT.


6. Hospital Nurse left...
11/14/2006 11:53 am

Some things from an in-hospital nurse:

1. KP-CIS does the job of keeping the critical information where it is needed and available to the bedside caregivers. EPIC does not and the constant downtime with patients whose lives are literally in balance means that everything gets printed out on EPIC making it less than useless.

2. It seems every loser from the healthcare side gets shunted over to HealthConnect instead of being fired. I'm not saying they are all losers, but the impression is that they are a dumping ground for performance problems coming from other parts of KP.

3. Charting is a licensure issue so even though we (bedside nurses and doctors) are a bunch of IT dinosaurs. We are not JUST a bunch of IT dinosaurs. When something goes wrong, it is our license on the line and I guarantee that, "the system was down", will not save our butt either in court or before the state board.


7. Gadfly left...
11/14/2006 2:32 pm :: http://corphq.livejournal.com

//So why is Justen so sure about facts and events from before he was at Kaiser?//

Just want to point out that Justen wasn't the only source for the ComputerWorld article. There was at least one other anonymous source as well. Also, I might be misremembering this, but didn't Wiesenthal (honcho of HealthConnect) also bag on Citrix in Matthew Holt's interview?

Also on Epic being pre-selected: I know from personal experience that people at the top of the heap (re: the medical group CTO of NCal) didn't know anything more than Halvorson was choosing between Cerner and Epic. Isn't Justen saying that it was Halvorson himself (and possibly some cronies like Dodd) who "pre-selected" Epic while pretending to go through a big decision-making process?


8. Anonymous left...
11/14/2006 2:56 pm

"By October, we were at 60,000 user hours of downtime and early in November we were already at 35,000 hours. It's a combination of poor network infrastructure and poor Citrix infrastructure. To say it’s all due to power outages reflects a poor understanding of what’s going on."

These numbers are meaningless. Quoting user hours without any context, number of users, or circumstance paints a slanted picture of what is going on.

It is worth noting that the Mid-Atlantic region had a planned downtime at the end of October to complete an Epic software upgrade. This alone would account for many of the user hours Justin has reported.


9. Denominator anyone? left...
11/14/2006 3:33 pm

35,000 hours out of how many? 60,000 hours in October out of how many? And the denominator is (drum roll) about 8,500,000 user hours / month.

If you're going to count every network drop, every wiring closet that takes a dive, every virus that trashes part of your network, every cable that gets cut by a backhoe, every computer or switch that crashes, every Windows server that locks up, you're going to get some big numbers in a system this big this widely deployed.

Maybe Justen and his disgruntled friends have some new super computer they're about to release and this is all just some marketing buildup for that? They sound like experts, they're treated like experts, just maybe :)


10. Bob LaBla left...
11/14/2006 5:09 pm

So based upon Mr. HIStalk's comment about wanting to learn from the KP situation, does the majority of HIStalk readers also want Epic to ultimately succeed at KP and other health systems where it is being implemented or live? Or, is bad press and implementation struggles too attractive to resist our attention and discussion? My opinion is that it would be good for all of us for Epic to consistently succeed at a number of installations. Looks like the guy from SCI is firmly in the Epic camp....how about the rest of you out there?


11. Citrix guy left...
11/14/2006 5:38 pm

This Justen guy is CLUELESS technically. KP is getting 70+ EPIC users per Citrix server, and there are many implementations with tens of thousands of users that are not KP. (Cingular) This is a dumb kid with a few facts and a ton of rhetoric, misinformation, and uneducated opinion.

How many Citrix servers are being used now?

"Thousands. We can only get 12 sessions per server, so that's several thousand.

We certainly went against our own engineering reports. Physicians liked Epic's feature set, but were concerned about scale. Their biggest implementation even today is 1/10 the size of Kaiser. We needed the billing module, but engineers said it wouldn't scale. George Halvorson even said in his e-mail that we needed to let Epic prove itself. That's not a prudent business decision and certainly the engineering reports have proved that. Kaiser's engineers are working night and day to support a system they said wouldn’t work."


12. datacomm guy (hospital nurse's h left...
11/14/2006 5:53 pm

Ok. Someone is seriously trying to mislead here. The uptime requirement for a system like this is 99.99%. The fact that Halovorson's response suggested the goal was 99.5% was the point at which any knowledgeable person knew that he was spouting BS. No mission-critical system has uptime goals so low. Customer service systems have goals higher than that (typically on the order of 99.7-99.9). Phone systems and credit card systems are in the 99.99 - 99.999 range so don't spout spin about the denominator when these numbers are an absolute embarassment when you're talking about a healthcare system. No wonder Justen was frustrated when this blindingly obvious fact about IT deployments is ignored (and defended by some anonymous KP stooge in the comments).


13. Misdenominator left...
11/14/2006 6:14 pm

To the poster who suggested 8.5 million user hours per month... It looks like you either have no idea what you're talking about, or you've based it on the number of Kaiser members. In either case, you're wrong. KP has somewhere south of 3.2 million user hours per month at this point. At the end of October, KP was seeing something like 98% uptime, based on user impact hours. I read that the first week of November was already at 35,000 user hours of downtime... That would put them on track for 95% uptime. Not acceptabled for a hospital, as Datacomm Guy pointed out.


14. Gadfly left...
11/14/2006 6:43 pm :: http://corphq.livejournal.com

If we use 8,500,000 potential user hours as the denominator, then availability is around the area of 99.0%. I wouldn't want to be the patient affected by that 1% or 1-out-of-100 downtime. And this is way below Kaiser's goal of 99.7% (which, as someone pointed out, is a low goal in itself).


15. Matthew Holt left...
11/14/2006 6:52 pm :: http;//www.thehealthcareblog.com

Datacomm guy has a clear point. As of now (from the data that I've seen which is the same as ComputerWorld has) HealthConnect is running in the 98-99% range. It is nowhere near the "five 9s" desired uptime other than in Colorado, and in some cases below 98%. So is that good enough? I tend to doubt it too.

The issue really is, what will it be when they've a) expanded it to the whole systems, and b) ironed out all the kinks.

Perhaps the obvious question is, when did Google or Ebay last go down and for how long? And (given that at least EBay had a huge outage in the early days) what do they know about reliability and back-ups that KP needs to find out?


16. GOODCITIZEN left...
11/14/2006 7:29 pm

Am I the only one concerned about how favorable these contracts are for the vendors? Epic has made a MINT--not just from Kaiser, but because of kaiser. Yet Kaiser stands to lose their shirt on this. Wasn't the original announcement a $1.8 billion deal (after writing off around $400 million)? Now it's $3 billion? And it's up (and unreliable) in 2 hospitals... What the f--- is happening? Is Kaiser already hemorrhaging because of this, or did I miss some other press release. If not.... More than a billion dollars over budget should at least raise a few eyebrows.

The entire IT governance committee + Halvorson should be canned. It won't happen but it should. Hospitals need to wake up and grow a set when we contract with these guys. If Epic won't negotiate and put some skin in the game, some other vendor certainly will.


17. Fly on the Wall left...
11/14/2006 7:32 pm

Will someone do some fact checking for crying out loud? Justen Deal is a Publication Project Supervisor in the Health Education and Training Department, not a Project Manager, and definitely not directly involved in the implementation of such a system. Does he even stop to consider that once running KP's HIT system will most likely be one of if not the largest private network on the planet? Its amazing to me to see so many people eat up his story so readily without cold, hard, facts! Most of what he says is pure speculation, such as "Epic will not scale" in his e-mail. How would anyone except some of the top systems analysts Kaiser has know anything about the issues surrounding the implementation of a system meant to hold so much information and be accessed by so many people in such a large geographical area? Why is no one asking questions!?!?


18. Techie question left...
11/14/2006 7:46 pm

OK, with all these uptime statistics being thrown around, can someone please tell me a good number for the implementation phase of a major IT project like this? 99.99% may be a reasonable standard for a system that has been in place for a while, but what about a system that is being rolled out and the bugs are being worked out? Is 98% reasonable then, or not?


19. datacomm guy left...
11/14/2006 8:41 pm

Attacking Justen for not being part of project implementation is just misdirection. The flaws here are so OBVIOUS that anybody can safely comment on this as a clusterf***. To the person who asked what is reasonable, it works this way in every deployment I have been involved with. You test the hell out of it in a systems lab environment with a test plan covering every possible scenario including having people pound on it with the intention of causing it to fail. Bug reports are created and classified by priority. No priority one bugs can be present for you to move to the next stage. Only priority two bugs with known causes and fixes on their way can be extant. Priority three bugs are fuzzier and are based on a subjective feel of the number and quality as to whether you can move forward with them. After system testing, you do load testing and it better meet the goal since this is a controlled environment. The load testing includes all elements including routers, servers, power, etc. Don't give me any BS about those things not being included in the uptime goal. At this point you go live at a beta site preferably in a dual mode although for this project I would guess that would be unlikely. No way you move forward from beta to controlled introduction (2 - 20 sites) unless you are still meeting every single one of your quality criteria. All this time you are collecting, collating, and learning. I can't tell you exactly, but each phase is measured in months so the load testing may be a real or simulated 3-6 months of use. Beta certainly is 2-3 months, if not longer; CI, the same thing. Finally, after you are convinced that this all works the way you want it to you go live. In the meantime, you are doing other stuff like focus groups with the users and maintenance to fix some of those priority 3 bugs that you let slide before beta. You don't have to be any kind of expert to condemn this implementation based on the published acknowledged FACTS so don't jump on Justen for not being an expert. It just shows you for a mealy-mouthed apologist, hypocrite, or idiot (you choose).


20. datacomm guy left...
11/14/2006 8:58 pm

Sorry, my comment should have been more succinct. No, 98% is not acceptable at any stage. You structure your roll out so you meet your uptime goals at every stage. The hope is that the numbers go UP (from your base requirement, in this case, 99.99%) as you increase the denominator and you gain experience (and you users gain experience). That's how I knew the commentor spouting about the denominator was full of it.


21. Anonymous left...
11/14/2006 8:59 pm

Epic's a good system as pointed out by folks who have actually worked with the system in other organizations and by KLAS reports. The scale of Epic at KP is new ground for Epic (or for any other vendor out there today is my guess) but they've been innovative enough to get around limitations with hardware and other technologies. When you have many moving pieces, all it takes is one part to fail to make it appear like the whole system is failing. I agree that 95% is too low of a uptime to be happy with and even 99.7% might be low. So many people are gloating at the failures (and perhaps perceived failures) of KP+Epic but BOB LABLA makes a good point that we should all be rooting for KP+Epic to succeed because it would be a good for everyone. Maybe Justen's motives were sound in wanting KP to have a good system but I think he's gone about it the wrong way. KP staff reading all this bad press about their organization must feel disheartened that all their efforts and successes are not being recognized because there are successes based on what I'm reading. Whether it's one hospital or all hospitals, every rollout, every user starting to use a EMR, every automation of a paper process is a success story in itself. I'm sure that prior to this fiasco and long after this has died down, the staff and management of KP have been and will continue to work hard to provide a good system and sound infrastructure for their organization and users.

It's easy to say "they should go with another solution", but unless someone is willing to stick their neck out and propose a different solution that they will know will work out of box and 100% of the time, I think too many people are just jumping on the "opportunity to bash" bandwagon.

Regardless of whether you're Epic or Cerner or Eclipsys, KP or UPMC or the 2 man doctor operation in the suburbs everyone involved in healthcare and HIT has noble goals. Be objective, listen to the facts from different sources, listen to the industry, don't take hearsay as truth ... don't fuel a burning match with a gallon of gasoline and claim that you were just trying to put it out to prevent a fire.


22. Anonymous one. left...
11/14/2006 11:04 pm

It is known within Kaiser that Garcia has initiated an HP-like information gathering program to identify employees releasing information to blogs.

Epic/HealthConnect does not comply with HL7 requirements for the distribution of electronic medical information. While initial record creation standards are met, subsequent record modifications are not. Any modification to a record results in the creation of a new record. This at best is curious, as Kaiser helped developed HL7 standards. One of the reasons for requiring that record modifications be treated differently than new records is to provide an audit trail for changes.

Citrix server farms run on Wintel platforms. A recent outage was cause by yet another Microsoft vulnerability.


23. datacomm guy left...
11/15/2006 12:19 am

Noone wants a successful implementation more than I. I am hospital nurse's husband and trying to explain the benefits of technology gets old pretty fast when she points to her grief at work from cra**y implementations. Let's not pretend, Kaiser's current implementation sucks. You don't have to be an EXPERT to see that so the person attacking Justen for not having credentials is as full of it as Halvorson. As Justen makes clear in his interview, it's not EPIC's or KP-IT's fault or even Citrix (although I'm pretty biased against Citrix from a basic architectural philosophy). It's a failure of leadership and management to the extent that they have lost sight of what they are trying to accomplish and have allowed a really lousy program to move forward by sheer inertia since noone apparently wants to point out the emperor has no clothes. Justen has done that and we can see the KP trolls and apologists on this site alone trying to argue otherwise. 99.5% uptime? Get real - that sucks. Billions over budget? Get real - that sucks. 100% Citrix implementation with no fat clients? Get real - that sucks. Money of the magnitude that KP is spending along with executive buy-in at that level should not result in faulure and does not have to even at this late date, but people have to be willing to acknowledge problems instead of bulldozing forward in the mistaken belief that acknowledging problems invalidates the program. I don't know enough about EPIC to agree with Justen about dumping it; I suspect that would be the wrong approach and that a collaborative solution with EPIC would be the right approach. On the other hand, I totally support him in publicizing a really awful situation that Kaiser was not willing to address because they believed it to be politically infeasible. He probably even got it right that the political raodblock was Halvorson who responded to his concerns with a BS broadcast e-mail.


24. Matt Helm left...
11/15/2006 1:24 am

Did anyone else notice in the ComputerWorld article that the KP-IT Quality dude is named Neidermeier? There is your problem right there.

Bluto: I'm not gonna take this! As far as I'm concerned... Wormer... He's a dead man! Marmalard... Dead! Neidermeier... Otter: DEAD!


25. Anonymous left...
11/15/2006 10:50 am

Datacomm Guy: The problem with Justen is not that he is pointing out problems at KP, the problem is that he is using the wrong information, opinions stated as "facts", and the wrong aproach. If you read his letters at fixkp.org, he is basically asking for the head of Halvorson for "forcing" Epic on KP, not for determining the reasons for the perceived problems, or measures to improve the situation. His letters to the board were for the investigation of how Epic was selected, and he got a response on that.

The assertion that "Kaiser was not willing to address" any problems with HealthConnect has not been verified, or shown to be true. Many people are fixated at what is going on at the hospitals, which are just starting to get the new system, and are forgetting that Kaiser's main business is primary and outpatient care, and that is where the system had been rolling out first. If the system outages at a hospital are that severe, this is what has to be addressed right away, but I didn't see any suggestions from Justen, except "OMG, the sky is falling, stop everything, and start over." Epic has technical solutions that can provide the information locally to the hospital if there is a system outage, I don't think Kaiser would have gone live without such a backup. When people are sayig the hosptial is falling back on paper, they may be addressing the use of printouts from the local backup system, which contains the latest clinical information available for the patients.

If anyone here believes that Kaiser and Epic are not working for resolving any system availabilty problems and are merrily pushing out rollouts just for the heck of it, you are misinformed. Justen has every right to have concerns about actual problems with the HealthConnect implementation, but he was not in the position to know what is being done, and how they are being addressed. His letters to the Kaiser board don't show inquiry into what is being done to imrpove the situation, they are laying blame for percieved improprieties, and present uninformed opinions as facts.


26. Gadfly left...
11/15/2006 1:26 pm :: http://corphq.livejournal.com

Has the KaiserTurf begun?

//Garcia has initiated an HP-like information gathering program to identify employees releasing information to blogs.//

Probably true - though Garcia ought to look at what KP is just posting on the Internet first. Still, it seems to me that this is the ideal place for the Kaiser Cronycrats to post a warning/threat. That way Kaiser saves the money, hassle, and bad PR of conducting an investigation.

//Justen has every right to have concerns about actual problems with the HealthConnect implementation, but he was not in the position to know what is being done, and how they are being addressed//

That reeks of KaiserTurf.


27. ClickRich left...
11/15/2006 3:54 pm :: http://clickrich.blogspot.com

Scaleability is complex. When someone says "it will not scale", this is actually rarely the case. The truth is that almost anything will scale if you throw enough hardware at it... but every extra box brings problems. I call Citrix the "necessary technology evil" because it enables many users to overcome software architectural limitations. Cerner implementations often use it too. The truth is complex here. Justin himself says that Epic is "perhaps the leader in the HIT sector". Who has actually proven scale out to so many users- 100k to 180k, depending on who you read, in this sector? At the end of the day, Epic is architected for many things such as responsiveness and user experience richness. I wouldn't have put scale at the top of their list, but then they're not entirely responsible for implementation architecture and running of the hardware.


28. Dr. Ethics left...
11/15/2006 5:53 pm

On KaiserTurf, hopefully Garcia has learned from Hewlett-Packard. Unfortunately, I suspect that Halvorson and Garcia are looking out for each other - which shows the inherent problem of a Board member also serving as the Chief Compliance Officer and involved in hiring the CEO. KP may be in a better situation if they had an effective Ethics program to support an open culture. I suspect Garcia's ethics officer lacks the will and clout to take on senior leadership and ensure the right things get done - such as ensuring there is no retaliation for reporting problems. Just by comparison, HP might have avoided much of the pretexting scandal if its chief ethics officer, Kevin Hunsaker, had asked more questions and demanded more answers instead of kotowing to his boss (the general counsel who resigned) and the HP Board. See: http://www.forbes.com/2006/10/23/leadership-ethics-hp-lead-govern-cx_hc_102 3ethics.html Doesn't Kaiser have a bloated ethics and compliance department? Last I heard there were upwards of over 60 people in the department and that's not counting indirect reports and related network structures. That's huge! And being headed by a member of the Board they still can't instill a better corporate culture?! What have they been doing?


29. Max Peck left...
11/15/2006 6:02 pm

I think Justen is calling it as he's seen it, and I won't hold his youth against his objectivity in the matter. KP-IT and planning seem to be polar opposites, how many contractors had to be let go early March due to a "slight" overage on the budget? Some had only been brought on weeks earlier, and to the tune of how many tens of millions? (talk was between 70-90) KP-CIS was the exact reason I choose to never work healthcare again, best to stay the course in the banking sector.


30. jd left...
11/15/2006 6:27 pm

Now that everyone is talking about uptime percentages, what is an appropriate percentage as a system is being implemented? Because that's the issue here. It's one thing to expect 99.99% uptime for an established system in which bugs have been worked out, it's another to expect it as a system is being rolled out. So what is the standard there?


31. Steve left...
11/16/2006 12:30 am

I think lost in all the furor is the fact that HealthConnect is working beautifully in the Kaiser Permanente regions where it has been introduced. Today, members in Colorado can e-mail their doctor, find their lab results online, check the doctors' notes from past visits, and see a list of their allergies. Similar features are available in Hawaii, Oregon, Ohio, Georgia, and the mid-Atlantic states. For several years in Colorado, patient records have been paperless, which has meant fewer errors caused by illegible handwriting, easier hand-offs to specialists, and just-in-time information in the emergency room. Colorado pioneered the CIS system that was later scrapped in favor of HealthConnect. I don't think you'll find many doctors here wishing they still had the old OS2-based CIS system. Outages happen, and we hope they will happen less frequently. But if the medical world waits until system outages are extinct before building electronic medical records, we'll remain in the 19th Century forever.


32. NotJusten left...
11/16/2006 10:00 am

Justen obviously knows very little about IT and less about Citrix software. He apparently doesn't even know much about what Citrix is doing on site with Kaiser. This money quote is like a billboard for Justen's lack of knowledge. "We're using (Citrix) in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network." This sounds like something that could have been said in 1998. It is certainly not true today. Justen should stick to working in Healthcare Education and leave Information Technology Project to people who actually know what they are doing.


33. Scott Chiara UC Davis Med Center left...
11/16/2006 5:02 pm

Here we use Epic and Citrix for 99% access to our Epic EMR system in a large environment. And can say for sure it works quite well.

Mr Deal has not communicated any solid facts or information about Citrix or Epic that indicate there is a problem with scalability on either.

I have been desgining architecting and deploying Citrix technologies since 1990, in large scale environments, to be used for internal applications access as well as external remote access. Citrix technology is very scalable and very stable if designed correctly.

Get real Mr. Real.

ScottC http://www.thn-world.com


34. I_was_there left...
11/16/2006 10:30 pm

Justen has his facts straight. He was very straight forward in letting us know he was not on the impementation team nor is he IT. Let's not attack him there. I live in a different region and was on the implemetation team. It was a fight every during build, validation and design to make KP mangement understand that Citrix was doing a job it was designed to do. The team repeately vetoed IT and numerous other presenters, to include the Kaiser Coalition of Unions, when we asked for changes. By no means should Citrix be to blame for KP's problems, unless you add every other vendor involed. The blame is with poorly written contracts with vendors, lack of communications within the KP infra-structure, micro-management of projects that lead to information loss and the large amount of siloed information in a build that should certainly remain transparent. The finger should be pointed at the management that ignored all that opposed it, choose to cut corners when thorough testing was necessary and spent KP money at least once a day to feed the group.


35. anonymous left...
11/16/2006 10:45 pm

I work for KP, and there's no way HealthConnect is up 99% of the time! The server is always busy. All of this denial is putting patients at risk.


36. Mike left...
11/17/2006 11:53 am

Truth be told: http://www.brianmadden.com/content/content.asp?id=639


37. m.m. Fisher left...
11/21/2006 7:07 pm

I work at Kaiser Sunnyside Hospital and I love Epic. It is the best for doctor,nurse and patient.


38. SA left...
11/22/2006 11:31 am

I recently left Kaiser after over 5 years in KP-IT. I left because the management chain all the way to Dodd could not effectively manage themselves out of a paper bag.

I did not work directly with the Epic project but observed daily the issues confronting the whole process. My biggest concern was that the consulting firm brought in for the implementation (DT) ran roughshod over all of KP-IT and management abdicated any control to them. The project was handled ham-handedly and timelines were considered over common-sense. And when the DT minions left, it seems alot of the learned knowledge left also. A few months before the implementation started, VP DW had stated that on any national projects that KPIT would take the lead in implementation and support for current systems would be backfilled. That was obviously just smoke. You would think you would want the people who were going to stay around to do most of the work.

I do know alot of the people in HealthConnect support for this region, and they are top notch. If there is any reason it succeeds it is because of them.

In my east coast region, the providers loved HealthConnect. I asked whenever I went to a facility for myself or to take one of my children.

I know in our Cerner Lab implementation we were getting just under 50 users per Citrix box.

Kaiser is a great health care system, KPIT needs to be revamped.


39. BeenThere left...
11/22/2006 1:38 pm

The worst thing about all this? Only the people in charge have all the data from all sides of the discussions...and they're not sharing/asking for help.


40. gadfly left...
12/02/2006 12:45 pm :: http://corphq.livejournal.com

While I agree with the assessment of KP-IT, I don't think the issue of IT project management on the business/medical group side should be overlooked. That's where a lot of the bungling is going on. At Kaiser, the main route to career advancement for physicians is management/administration, especially IT management. There are too many physicians with a vested-interest crowing for "physician oversight" in order to create leadership positions for themselves and not enough people thinking about whether these people have the necessary experience/qualifications to lead IT projects. In my experience, a lot of physicians *think* they know more about IT than they really do. They should stick to UI testing - and even then keep in mind that their judgment might be tainted by lack of experience.

The ultimate tragedy here is that the big IT Leadership paychecks that are being wasted on ineptitude come out of money that could be used for patient care and lead to skyrocketing membership fees. "Technology Investment" = "Covering Up For the Last Mistake We Made."

I also totally agree that the worst thing of all is that the people with the biggest vested interest in covering up for Kaiser mistakes control all the data. That's been my complaint about Kaiser all along - abuse of control of the evidence infects all levels of that organization, and both patients and employees are regularly damaged by Kaiser's approach to risk management. The first way to support dissent and break groupthink is to give open access to the evidence.


41. Anonymous left...
12/05/2006 8:38 pm

As a Kaiser Permanente technical staff member, I am very familiar with the extreme technical deficiencies prevalent within the organization. I am also sorry to say I too have first hand knowledge of the draconian tactics employed by certain egomaniacal leaders and their selected minions.

Mr. Deal has correctly identified many of the concerns we face as Kaiser Permanente employees and patients. Although some very influential and key Kaiser Permanente executives are intimately involved or have first hand knowledge about the issues brought forth in Mr. Deal's statements, they have instead elected to become good corporate residents and hide the truth. As with other recent historical events, technology can assist in lighting a path directly to the source of the corruption. Kaiser Permanente’s flagrant disregards of technical requirement will also be their downfall because they can no longer quickly plug the technical holes they have so apathetically allowed.

I highly recommend for Department of Justice to adhere to their established strategies and begin their investigation by following not only the money but also the influence and destruction of evidence that has and is currently occurring. To succeed they should nevertheless consider offering protection and amnesty to potential witnesses and begin their investigative focus on the decision cogs of the organization, i.e., the CFO, the Board of Directors, the C-Level Executives, Internal Auditor, External Auditors.


42. Anonymous left...
12/07/2006 3:26 am

I can clearly remember the shock that went through our area when we learned about the seemingly abrupt Epic decision. As we were learned more about the system we became worried. The early information talked up Epic's success in small med centers. According the the Epic head, all it had to do was "scale up". Yeah, right. Another of her points was that few changes were expected. Yeah, right again (if you know anything about Kaiser). It was worrying when we learned that we were targeting roll out of the replacement for the application I was working on at the time to start in six months and that upper management wasn't interested in discussion. As my manager reported from a meeting with her peers with their manager, the word from above was "Believe or Leave". A lot left.

Curious, in the early days we heard a lot about the personal relationship between Halvorson and the head of Epic; how he mentored her when she was a student at the University he served at and how he assisted when she was starting Epic. No word of that now. I believe he as able to get Epic partially installed in the small med center he headed before he ran away from an investigation. Still a partial install, I believe.

So, now Kaiser is burdened with a system that requires six instances just for NorCal (reached its scaling limits, hard!). The intercommunication between the parts had to be quickly built as it was new to Epic. The application systems I worked with initially just sent transactions to Epic where they would then be rerouted to wherever instance the relevant records actually resided in (they can move). Opps, Epic can't handle the internal transaction load between instances and the instance receiving the most transactions is being overwhelmed by them. More outages/doctors wondering which instance they need to log onto to see the patient data. Now the external applications use a complicated set of rules to guess which instance is probably the correct target for each transaction and send it directly there. Not exactly a black box.

Oh, and the prior poster blessing Epic for its wonders on 'large med centers' such as UC Davis Med Center has no idea what "large" means.

Let's see 1.8 billion over 1.5 years.. Nope, better make it 6+ years (2009 completion) at 1.5 billion each plus an allowance for the greatly increased continuing overhead. Guess the estimate and timeline need revising. Better raise the rates while we're at it.

I hope Kaiser can pull it off, but there's a long way to go with this assemblage of independent green-screen applications glued together is a pasted-on gui.