As a software quality assurance professional with over 15 years’ experience in the field, I was initially shocked by the technical problems with healthcare.gov. I’ve devoted a large portion of my professional career towards ensuring that things like this never happen to my customers! However, after much thought, I realized that the ramifications of bad quality are quite different for healthcare.gov than they are for any commercial sector application. First, healthcare.gov has no competition, so user experience takes a backseat to deadlines. People HAVE to use it, unless they want to defect to Canada. Second, what lost revenue is the government seeing from the healthcare.gov issues? If anything, they’re getting free advertising and they will gain revenue whether or not people use it (either from those getting their insurance through the exchanges, or those paying the tax for not doing so). On the other hand, competition is all but assured in the private sector and rapidly emerges in those rare areas where it is scarce. In addition, almost all negative experiences in the private sector have revenue implications. Therefore, can I really use my experience in the commercial space to pass judgment on healthcare.gov? Absolutely!
All politics aside, these issues come down to a project plan that did not allow for adequate time to test for functionality, performance and security. Whether that testing occurs throughout a Continuous Delivery pipeline in an Agile methodology or at the end of the process in a traditional waterfall approach, it simply was not done. According to the Washington Post, only 2 weeks were allotted for full integration testing. And the National Review has reported that performance testing showed that only about 1100 concurrent users could be sustained. Of course, we have no way of knowing what the original requirements or user stories were (assuming they ever existed), but by any logical reasoning, this system could not sustain the enrollment of several million people. Just looking at the math for load levels, healthcare.gov went live on Oct 1, 2013 and open enrollment ends on Mar 31, 2014. That represents a 182 day window to get all Americans requiring insurance enrolled in the new plans. Now, if we restrict the number of Americans that need to purchase insurance through healthcare.gov to only those currently without insurance (not counting those whose insurance policies are being canceled or those registering and not purchasing), the latest estimates put that number at roughly 48 million people. Quick division now tells us that an average of 263,763 Americans will need to purchase their insurance each day throughout the open enrollment period to meet the Mar 31, 2014 deadline. Making the rather shaky assumption that it takes an average of 30 min to complete the insurance enrollment and purchasing process, we now have an estimated average concurrent load of 5494 users at any given time. As you can see, this is a far cry from the 1100 concurrent user maximum load previously referenced! However, as many corporations have done in the past, the system was promoted to Production anyway, with the inevitable “fixes” made on the fly (in Production, where they are the most expensive). I do believe that healthcare.gov will eventually “work,” but at a cost of time and frustration that was wholly unnecessary.
Application security appears to have suffered as well from this lack of focus on quality. Although many commercial sites are required to undergo static analysis to identify potential vulnerabilities before deployment, it appears that the same standards were not used in this case. In fact CNBC has reported that David Kennedy, head of computer security consulting firm TrustedSec and a noted security expert, prepared a briefing for Congress testified that there are application-level vulnerabilities live on the site as recently as Nov 19. The House Oversight and Government Reform Committee released a memo stating that “the threat and risk potential is limitless,” including the potential compromise of Americans’ personal information. I am happy to report, however, that a top Homeland Security Department official has reported that a DOS (denial of service) attack was unsuccessful. Of course, since a DOS is little more than an unauthorized performance test, this finding is not terribly shocking.
Now, back to the question of whether I can use my experience in the commercial space to pass judgment on healthcare.gov. I will answer that by telling the tale of another high-profile organization, this one in the commercial sector. That organization had a similar issue, which occurred annually on what should have been their biggest day of business. Each year, like clockwork, the users would flock to their site, clamoring to order the goods and services that this organization provided. And every year, like clockwork, that site failed to scale and was soon offline – unable to deliver to the people what they so desperately wanted. However, this organization eventually became serious about quality and their ability to handle all customers who wanted their products, even on that fateful day each year. Because of this, my colleagues from Orasi were called in and one of the largest performance tests in our history was performed. Many issues were found and resolved before that fateful day again rolled around, and this organization began to believe that this year they would weather the storm. Then the momentous day was upon us all and, in spite of major issues that impacted others, this organization stood proud and served every customer for the first time because they put emphasis on quality and their faith in the results that Orasi provided. Yes America, you may not be able to enroll in healthcare.gov at this moment, but at least you can rest assured in your ability to order pizza securely during the biggest day in professional football.
Bill Hayden, Vice President of Resource Enablement at Orasi Software
