How The COVID Tracking Project scaled to handle 200M API calls per day
March 7, 2021 marked the one-year anniversary of The COVID Tracking Project and the end of a year-long volunteer effort that has provided critical information on COVID-19 testing to hundreds of newsrooms, research projects and even the White House itself.
Back in April, when we spoke with Kevin Miller, the web design team lead, the volunteer team saw the project as a temporary measure until the government was able to take on this critical task. No one expected The COVID Tracking Project site to become the source of testing data for the duration of the pandemic.
In that year, a site that was pulled together in less than a month, scaled from 0 to 200M API calls per day, flawlessly handled traffic spikes and expanded to include data on race and ethnicity and long-term care facilities.
How it all began
On Feb 29, 2020, the CDC stopped posting COVID-testing data on their website. This raised alarm bells for people who knew how critical that data was in understanding the spread of the coronavirus in the United States. Two journalists at The Atlantic, Robinson Meyer and Alexis Madrigal, decided to go after the data themselves. Separately, Jeff Hammerbacher, founder and general partner at Related Sciences, decided to do the same.
On March 7 they joined forces with managing editor Erin Kissane to launch The COVID Tracking Project at The Atlantic.
They needed a fast, stable website
At this point, Meyer, Madrigal and Hammerbacher were manually collecting data from health department websites across the country and entering it into Google Sheets. Because there was no standard for reporting testing data, the team couldn’t simply scrape it. They needed people who could interpret footnotes, changes in definitions and metrics to make sense of 56 non-standard data sets.
Kissane put out a call for volunteers, including an urgent message to her friend Mat Marquis. Could he build them a fast, stable website? Within four days they launched the site on Netlify. Less than a month after that first version, they launched a more accessible version with a stack composed of Netlify, Gatsby and Contentful.
A modern technology stack empowers builders
Continuing to update and expand the testing data available on the site was a complex undertaking that involved managing hundreds of volunteers remotely. These volunteers updated the site about 300 times per week and rebuilt pages dozens of times a day, often juggling their efforts between family and work obligations. They didn’t have time to contend with a complicated interface.
Miller had worked with Contentful in the past and was familiar with the Contentful web app, which is designed to make life easier for content creators. The volunteers were quickly able to use the web app to collaborate on and publish the data, despite differences in time zones, locations and tech knowledge.
The team managed and updated the site with Contentful, using a Netlify integration that enabled the team to preview and develop pages together. They chose the GatsbyJS static site generator to handle large sets of data changes without formatting issues.
At Jamstack’s virtual conference in May, Kissane told attendees that the website had scaled from 0 to 200 million API calls daily — in just three months. When the White House cited the project data in their reopening plan, site traffic surged to a level that normally crashes retail sites every holiday season. Kissane said the team didn’t even notice — there were no slowdowns, crashes or overage fees.
"Having our stack be so solid, so stable and so fast has been a saving grace in the middle of this otherwise extremely complex project."
She added that she doesn’t think this would have even been possible without a modern technology stack.
At Contentful we believe the future is all about empowering builders to do great things. The COVID Tracking Project is testament to what people can accomplish and how those accomplishments can be amplified by technology.
In How digital experiences powered by Contentful are impacting communities in need, we talk about how other non-profit groups are making the world a better place.
