The vaccine software is one giant 404 error
Government software is stuck in the 1990s, and now we're all suffering for it.
Here we are, almost exactly 3 months after the first dose of the Pfizer vaccine was administered in the U.S.
And we’re delivering more vaccines per day than ever before! Progress is definitely being made. But we’re slowed down by terrible technology at every level.
A combination of entrenched government contracting practices, outdated software, and general intransigence has almost certainly cost lives due a slower rollout – and of course, those with lives at risk tend to be those who are already suffering from the worst of government technology systems.
Back-end vaccine technology
My takeaway from my research is: the back-end technology isn’t as bad as it could be. Here’s how it works.
First, vaccines are released by Pfizer, Moderna, and J&J. Each company tries to give HHS advance notice of how many doses will be available in that shipment, but the number isn’t always precise, especially now in the early days, when each company is actively working out the kinks and making their manufacturing technology more efficient (a process that usually takes place before the vaccines are publicly available, but which is happening alongside the rollout in this case because everyone is moving so quickly).
Once a confirmed number of doses becomes ready for shipment, HHS inputs the number of available doses into software called Tiberius, created by Palantir. Tiberius uses Census data for individuals aged 18+ to determine how many doses should go to each state or territory.
As MIT Technology Review reported in a very helpful overview of the back-end technology in January, Tiberius can be programmed to distribute on the basis of seniors 65+ and, theoretically, the number of individuals with health conditions that would put them at greater risk of COVID-19 complications (although the latter would require an overlay of integrated health care data which…does not exist). But for right now, it appears that doses are being distributed on the basis of adult population.
Tiberius then notifies states/territories of how many doses they can expect, and allows state/territory officials to overlay demographic data, the CDC’s social vulnerability index data, and the location of subarctic storage freezers to determine where within the state/territory to send doses. (In January, Palantir also added an exchange feature to Tiberius, allowing states to swap vaccines that have different storage requirements as needed.)
The dose count data simultaneously goes into files that employees of the CDC manually download and input into another technology tool, VTrckS. MIT Tech Review describes VTrckS as “a legacy system that has passed through multiple vendors over its 10-year existence,” and which is “eons apart technologically” from the relatively well-oiled Tiberius software.
VTrckS acts as an online ordering portal for states to order the number of doses for which they’re eligible, and to tell the manufacturers which sites to send which doses to. Once states upload the desired vaccine count per address, that data is routed through the CDC and on to manufacturers.
Next, the shipments take place. Despite my earlier concerns about our national supply of dry ice, it appears that the shipping process is perhaps the smoothest part of the entire rollout. Finally, all of that data feeds back into Tiberius (see graph above) to help inform the distribution of the next set of doses.
Front-end technology
At the other end of the spectrum, the front-end software used for vaccine appointment scheduling is the worst part of the vaccine rollout.
There are a few reasons for this. First, state and local governments are routinely underfunded. Many don’t have a large budget for technological investment, and some were forced to spend money they hadn’t necessarily budgeted when the CDC’s vaccine ordering software proved a nightmare (see more about that below).
Second, state and local governments have little incentive to apply the funding they do have to making software better. This is both because there is no market incentive to do so, and because most state and local software is used for accessing benefits and food stamps, typically by people who don’t have enough of a voice in politics. In other words, state and local governments have gotten away with terrible software for years, but because this software had been forced on more marginalized populations, most people didn’t notice or care.
Third, many government offices have little or no technical expertise, meaning they lack the ability to push contractors to create more user-friendly or high-capacity products.
And finally, the government contracting process is a mess. More about that below, but the gist is that outdated rules prioritize big companies over more innovative ones, and the contracted companies have limited incentive to work hard, well, or fast. Instead, the contractors can receive more compensation for taking a longer time, and building something that breaks—so they can then be compensated for building a fix.
The CDC, for example, paid Deloitte $44 million for a no-bid contract to create a platform that could be used for ordering, scheduling, and reporting vaccine doses. The agency planned to distribute the platform to states for free, so individual states didn’t have to handle contracting for a site on their own.
But the resulting platform is terrible. The head of South Carolina’s health department told South Carolina state lawmakers that the site, known as VAMS (Vaccine Administration Management System), has “become a cuss word.”
Similarly, DC relied on Microsoft to create its website. The resulting site is such a mess—littered with obstacles like a broken CAPTCHA, a complicated form, and a map showing possible vaccination sites without showing which sites still have supply—that Microsoft released a statement acknowledging that “our efforts have fallen short.”
A friend of mine had a different take:
DC recently decided to try a different system, one with a lottery format, instead of a timed rush for appointments a la concert tickets. They, again, went with Microsoft, although the result remains to be seen.
No user friendliness
Something I’ve been thinking a lot about recently is the concept of user friendliness. In a book called User Friendly: How the Hidden Rules of Design Are Changing the Way We Live, Work, and Play, authors Cliff Kuang and Robert Fabricant talk about the history of user friendliness as a concept.
During World War I, for example, the first large-scale industrialized war, soldiers were suddenly exposed to complicated machinery with little training or experience. The machinery, in turn, had not been designed with soldiers in mind. Radar screens were counterintuitive, planes were challenging to fly.
Since then, designers have gradually centered user friendliness as more of a mandate. The motions used to navigate an iPhone, for example—swiping up to access all your open programs, or using cellphone features familiar from analog phones—are intuitive for most users. User friendliness, in other words, has become an essential part of technological design, particularly as technology permeates more areas of our lives.
When using technology for an essential purpose, incorporating user friendliness isn’t just about enjoyment of use. Unnecessarily complicated technology can stymie anyone, but it becomes particularly onerous on older people or those with other limitations. People on a library computer with user time limits, for example, might not have time to click through pages and pages of forms. If a website isn’t easily decoded by screen reading technology, blind users are unable to use the site.
All of these factors are coming into play with the vaccine sites.
As Kaiser Health News reported, many vaccine registration sites at the federal, state, and local levels violate disability laws. VAMS, the Deloitte-created site, seems to be at the root of much of this, but state and local websites also failed to consider the challenges of access for blind, disabled and elderly users, let alone those without easy access to the internet.
As a pediatric urologist in Connecticut told MIT Tech Review:
[VAMS] won’t work on Internet Explorer; it only works in Chrome. The ‘Next’ button is all the way down and to the right, so if you’re on a cell phone, you literally can’t see it. In the first round, people using VAMS mostly had advanced degrees. If you’re 75 and someone asks you to log into VAMS, there is zero way it’ll happen without help.
Government procurement challenges
There’s a lot more to say on the government procurement challenges front, but I’ll offer a summary largely lifted from the work of Hana Schank, the Director of Strategy for Public Interest Technology at New America.
In a Fast Company article published in April 2020, she writes about how government tech is stuck in 1993. “Of course the state unemployment systems are written in a 60-year-old programming language. Of course there’s no way to find out the status of anything you’ve submitted, ever. Of course the sites are crashing,” she writes. There’s been no market pressure for change, and no commitment to user experience.
And, as mentioned above, the government procurement process is driven by familiarity over innovativeness. Government regulations may require a particular contract go to a company with a history of other federal contracts. This excludes smaller or more innovative companies which might be able to do a better job.
Conclusion
So here we are. It’s frustrating to be near the end of the pandemic, to have increasing numbers of shots rolling off the production lines, and to be stymied by a Microsoft site stuck in the 90s. But it’s the fruit born of the way governments request, contract, and project manage software.
On one hand, the public-private partnership that was Operation Warp Speed (OWS) was a technological marvel, producing vaccines far faster than even the most optimistic observers would have guessed a year ago. Breathtaking research for the benefit of humankind coming out of such partnerships is possible.
On the other hand, the close federal attention, funding, and expertise that characterized OWS has been absent from the vaccine distribution technology process, and it shows. This part of the rollout is slow, clunky, and littered with 404 pages.
Until we deal with outdated government software and the practices that brought us here—or demand that public-private partnerships look a lot more like the first part of the rollout than the second—we’re stuck hitting refresh and hoping Internet Explorer is on our side.
This was my favorite article yet! Loved the behind-the-scenes info. I thought your thesis was powerful — it really hit a nerve in me. So much more work is needed to foster rapid innovation.
“ A combination of entrenched government contracting practices, outdated software, and general intransigence has almost certainly cost lives due a slower rollout – and of course, those with lives at risk tend to be those who are already suffering from the worst of government technology systems.”