SOASTA specializes in scale and performance testing to nip problems with websites in the bud before launch. Able to simulate the simultaneous clicks of many millions of visitors, it's been enlisted by the websites of the 2012 London and upcoming Sochi Olympics, NASA's Mars Curiosity rover program and "pretty much all the major e-commerce brands these around the world, companies that do billions of dollars every year through their websites," says Lounibos.

He too is baffled by the Healthcare.gov site's many failings so far.

"Intuitively, for all of us consumers, it does seem like a logic test, and everybody's flunking," he says.

'We can do better'
Even the unceasingly energetic and optimistic Todd Park – known to most healthcare IT observers as the cofounder of athenahealth and the former chief technology officer of the Department of Health and Human Services – sounded frustrated when he spoke to the New York Times earlier this week.

These days, Park is CTO of the United States. The site, he said, was a victim of its own success. "At lower volumes, it would work fine," Park told the Times. "At higher volumes, it has problems."

"Right now," he said, "we've got what we think we need. The contractors have sent reinforcements. They are working 24-7. We just wish there was more time in a day."

The newspaper also spoke to Park's predecessor as CTO, Aneesh Chopra. He was the first man to hold the position Obama established in 2009.

"This is par for the course for large-scale IT projects," Chopra told the Times. "We wish we could launch bug-free, but in reality that's not that easy to do. The reality is that if you have a product that people want, people will tolerate glitches because they expect them."

Still, it seems to defy belief that such a hugely important and anticipated site could be rolled out so sloppily. Its failure has chagrinned Health and Human Services officials, who have rushed to make big improvements to its site design and infrastructure.

"We can do better and we are working around the clock to do so," HHS press secretary Joanne Peters told the Wall Street Journal.

This all could have been avoided, Lounibos tells Healthcare IT News.

"There's really no excuse," he says. "They've had enough time, from what I can tell, to fully test. Our retail and e-commerce customers continuously test – not waiting until the end to test but literally testing every day, all the time, new code that's being generated. It doesn't appear that was that was done here.

"It's amazing how folks miss the best practices," he says. "And a lot of it – easy even for those of us who are not engineers to understand – is the issue of speed versus quality. Developers that are building websites, even if they have a long run time to get to the launch date, are rapidly developing and adding features and functions. Every day they're trying to reach a milestone – in this case a roll-out to the live audience – and they're up against a deadline and they have a lot of things to do and usually the last thing that's done is testing. So quality goes out the door and speed takes the advantage."

And sure, a lot of people came to the site at once. But shouldn't that have been expected? At any rate, that's something that could and should have been tested for, too.

"No one really can predict how many users will come into a site, what we call 'load,' but the tools are available to literally recreate millions of concurrent users hitting a site. So there's really no excuse for that," says Lounibos.

"There's clearly a break-down here. It might be at the design level, it could be at the simple testing element, where they tested it one day, did some more code build and then didn't test those changes. It could be that they built the application and then never tested it on the infrastructure, the servers that they were going to use, and it might have been a configuration setting that might have broken."

Certainly, the average website for a hospital or a medical practice will never experience as much traffic as Healthcare.gov. But there are some lessons they can take away from this, best practices to keep in mind when launching new sites, says Lounibos.

"There's three of them," he says. "First, do continuous testing. Test through the development cycle. Test the first day, and test everyday.

"Second, test across environments. Make sure you're testing the back end and the front end together, and not separately. A lot of the problems occur at the connection points – the cartilage to the bone, so to speak."

And finally, "Make sure you do load testing all the time," he says. "It doesn't have to be millions of users. A lot of problems will occur with 100 users on a site. You just have to do some sort of load testing and you have to do it consistently."

In general, "just recognize that you've always got to take a quality first initiative, as opposed to speed," says Lounibos. "And ensure that the user experience – especially when you're trying to enroll people, as you are here – is easy and simple and fast. The user experience is the whole gig. You want to make that experience as painless as possible."