Delivering on the DOD Data Strategy with DataOps

The Defense Department’s Data Strategy delivers a clear mandate: Data is a strategic asset that will be operationalized to provide a lethal and effective Joint Force. With this new mandate in hand, the next big question becomes: How do agencies make the change?

By focusing on just 20% of the objective -- DOD Data Strategy Sections 2.2.4 (Data Collection) and 4.3 (Make Data Understandable) -- DOD could produce more than 80% of the intended objective results.  

Data Collection and Making Data Understandable are in the critical path of getting data to mission commanders, warfighters and decision-makers in a real-time, usable, secure and linked manner. In a recent interview with Defense Systems, Air Force Chief Data Officer Eileen Vidrine drove home the importance of high-velocity data interoperability: “We have a data strategy that's codified. It is the scaffolding, the guidance to help us. It clearly and specifically aligns to the [National Defense Strategy]. The concept around our visible, accessible, understandable when trustworthy, interoperable and secure [VAULTIS] -- interoperable is critical as we work across the Department of Defense. And when we start having great high-end quality data, that's consistent across our department that we have confidence in, it's going to build our trustworthiness.”

Data Collection and Making Data Understandable are among the toughest challenges because they’re historically labor-intensive and complicated by human problems like data hoarding or (in the case of government) “robo-classifying” data.

Automating these two processes is a foundation of DataOps, a business methodology that brings data and operations together to speed success. Already in adoption by the commercial sector, DataOps is basically DevOps for data. Like DevOps -- which moved from waterfall to agile development methods, saving time and money and improving quality in delivering software -- DataOps uses more-agile data management based on defined architectural principles to speed data delivery and quality. (See Figure 1.)

DataOps builds out the new generation of data infrastructure and human behavioral norms for managing data as a strategic resource on par with weapon systems. Using modern data unification methods, data is managed as a common, usable asset and shared over established data pipelines to critical consumers, in a timely manner.

Data unification is particularly essential in DOD because the quality of data will determine the quality of decisions made on the front lines for the operational community and the warfighter. By using a strategic combination of artificial intelligence, machine learning (ML), humans and management technologies, DataOps can shorten time-to-quality of data, collating and correlating all component data sources to power accurate analytics and decisions. 

For DOD missions, the result is information that is up to date, accurate and unified across a myriad of sources for more granularity. Data unification involves automating entity resolution, record deduplication and classification of lists of information into a desired enterprise-wide ontology.  

In the area of Data Collection, DataOps is used to automate schema mapping (to deliver usable data models), record matching (to create “golden records”) and categorization. In the area of Making Data Understandable, it involves tagging, storing and cataloging data. Intelligently automating the manual work behind these two processes using AI and ML will reduce cycle times, improve data quality and speed time to mission.  Eventually, the level of automation will be such that when random datasets come in, they’ll be immediately understandable. No longer will data disappear into a proprietary contractor database, never to emerge again.

All of this can make AI algorithms, as well as any analytics an agency may need, much more effective. Operators and military decision-makers will be able to harness data for strategic and tactical advantage on the battlefield, while senior leaders will be able to make better decisions and data scientists can deliver better analytics.

When based on a good data unification strategy, DataOps can help DOD agencies:

• Improve readiness and reduce the cost of aviation maintenance and repair: By creating a common ontology DOD can power predictive preventative maintenance.
• Speed up flight missions: By automating the process of understanding and providing searchable context to 40 years of test flight data, data teams are facilitating inbound requests for review of new aircraft configuration through automated discovery of relevant antecedent flight tests.
• Increase tactical awareness: By using and correlating spatial-temporal, feature-rich data and other data collections from multiple, advanced sensors observing common areas of interest, DOD can boost situational awareness.

Individual, idiosyncratic data silos -- a problem in spades at DOD -- creates a large gap in institutional knowledge, making it hard to find, access and understand the best data for critical business and mission questions. It’s routinely observed that 80% (or more) of data-handling effort today goes into data preparation, leaving only 20% for data deployment and analysis. 

The DOD Data Strategy needs to flip those numbers. DataOps can help.