How COVID-19 Kicked a USAF Software Team Into High Gear

The COVID-19 crisis has forced operational changes at every level of every organization. But in some situations, like that of the Air Force’s internal DevOps platforms team, that disruption has pushed them to new heights.

In March, the Platform One team—housed within the Air Force’s Office of the Chief Software Officer—achieved the gold standard in agile software development: deploying 10 software updates a day for a single platform, and sustaining that rate over time.

The Platform One team launched in January to help developers across the Air Force get the platform support they needed. While the Air Force has other innovative software development programs like Kessel Run, those focus on the end-users and mission owners. The Platform One team is here for the developers, creating a “DevSecOps-as-a-Service” environment, the program’s director, Maj. Rob Slaughter, told Nextgov in an interview.

The team is currently working on a diverse set of Air Force programs, from major, locked-down weapons systems to “party bus” platforms where it’s the more the merrier, Slaughter said. The list of programs includes the F-35, F-16 and F-18; the Joint Artificial Intelligence Center; Northern Command Pathfinder; Space C2 and SpaceCAMP; the Air Force Research Lab; Army Futures Command; and many more.

Related: The Air Force's Kessel Run Is the John Travolta of Defense Acquisitions

For the more sensitive programs—like weapons systems—the team builds custom platforms that can integrate with those complex systems and give an added layer of security. That said, security is a focus for the party bus platforms, as well.

The team has secured authorities to operate—a cybersecurity standard that certifies a product or system meets a baseline of security—that are continuously monitored and reauthorized. The agile nature of the team’s work also increases security, Slaughter said, as daily iterative development allows you to fix security concerns and deploy patches more quickly.

Slaughter said he has been preaching the virtues of DevOps across the Air Force for some time but didn’t realize how far his team was from true agile development until the COVID-19 crisis hit.

As the majority of the workforce was urged to work from home, the Platform One team was charged with deploying a new, secure communications platform. That secure platform became a “dire need” as tens of thousands of Air Force personnel began working remotely. 

The Platform One team was able to go “from zero to available and accredited in 48 hours,” Slaughter said. But as more people logged on, the strain on the system required constant updates. 

“The only way to keep the system up and alive was live in production,” he said, forcing the team to deploy code, assess and redeploy with haste, and with minimal to no downtime.

By March 18, the team had gone from deploying code once a week—already impressive in the defense space—to hitting the 10-or-more-deploys-a-day mark.

Don't miss:

That metric comes from a 2009 presentation at the O’Reilly Velocity 09 conference by John Allspaw and Paul Hammond, then head of operations and head of engineering, respectively, at Flickr, which had achieved that deploy rate before DevOps became a thing. Since then, for some developers, if you’re not deploying more than 10 times a day, you’re not really doing agile development.

“We’ve consistently been at 10 deployments per day since that occurred,” Slaughter said, noting the pace has dipped below that threshold at times. “For a while there, we were in the area of 30 deploys per day. Now, we’re probably at, like, 12.”

Slaughter said that deploy rate was always the goal, but it was largely aspirational until recently.

“COVID forced us to accelerate our journey,” he said, and showed the team that 10 deploys a day is possible, even in a military environment, even in the midst of a pandemic. Further, meeting that baseline is the only way to have a true agile software pipeline, he said.

“The community is lying to itself to say otherwise,” Slaughter said. “And months ago, I would have agreed.”

Most of the current Platform One team—including Slaughter—have experience working on app development in wartime, when a timely code deploy could mean the difference between life and death.

While the current enemy can’t be seen or touched, Slaughter said the pace and urgency of operations are the closest he’s ever gotten to feeling that pressure during peacetime.

“The department is always postured for defense. So, whether it’s officially wartime or not, the importance and the urgency is sort of omnipresent,” he said. “How people are supporting the warfighter changes depending on what your job is, but the underlining, key capability is communication. If you can’t communicate with people, then you can’t fight a war.”

The deploy rate for the communications system has since slowed, Slaughter said, as it doesn’t need as much maintenance. But his team has shown they can meet and exceed the 10-a-day threshold. Now, that capability needs to take hold throughout the military.

“This is the beginning of a long journey of total and complete transformation within the entire Department of Defense,” he said.