How big data is paying off for DOD

Analysts can monitor more inputs when computer analysis highlights potential areas of interest.

The hard part, of course, is making good use of all the information that’s becoming so easy to collect. And although no one claims to have completely solved the big data puzzle, the Navy, like the other services, is finding ways to take advantage of the power of analytics.

At a basic level, big data takes over the grunt work of sifting through reams of information that would be just about impossible for humans to handle. Networked systems pull together input from a large number of sensors on unmanned aerial vehicles and satellites along with many other data sources. Humans can analyze some of this information, but there’s more information to be mined from the thousands of gigabytes being continuously generated. Computer analysis can handle mundane tasks, freeing humans to do chores that better fit their talents.

“Big data is a force multiplier for the folks making decisions,” said Thomas “Lee” Zimmerman, head of the Communication and Networks Department at the Space and Naval Warfare Systems Command’s Systems Center Pacific (SSC-PAC). “People are good at recognizing patterns and making judgments, but they’re not as good in rote tasks like looking for tiny ships on a huge ocean.”

Many of the Defense Department’s big data projects aim to make the best use of unmanned aerial vehicles. Some are designed to help analyze files, while others aim to use UAVs more efficiently for intelligence, surveillance and reconnaissance (ISR) work, which at times can come down to something as simple—though potentially huge—as storage.

“One challenge with UAVs is the vast amount of data they collect,” said Christopher Raney, head of SSC-PAC’s Command and Control Technology and Experimentation Division. “Video is often dropped on the floor; it’s not stored. If you put it in the cloud, you can save a lot of time. Widgets can be installed on UAVs to store select information.”

This sequence shows a data cloud growing from UAV inputs--more than 10,000 inputs in 17 seconds.

He explained that when UAVs are scouring the seas for mines, for instance, a UAV operator and analyst typically work together. If the analyst is unsure about an area and wants to double-check it, the controller will have to send the UAV back over that section again. But if that data were stored, that section could be flagged for later review and the UAV could continue its planned flight path.

Data mining software can also spot objects of interest so analysts don’t have to personally scour huge images. Finding ships at sea is a primary example. Computers can quickly search for items that are likely to be ships. If these programs are put on smart sensors in the air, the volume of data will be reduced, freeing up bandwidth for other types of sensors.

Software can pinpoint ships so analysts don’t have to scour the seas to find vessels

Targeted information

As more of this defense data is stored in the cloud, keeping it secure becomes a greater challenge. Making files accessible to most or all warfighters also makes data more of a target for foes. New technical solutions are being used to ensure that adversaries don’t alter data.

“From a cyber perspective, can adversaries inject disinformation into our systems?” said Peter Tran, senior director of RSA’s Advanced Cyber Defense Practice. “Counter analytics look for disruption, for example looking at UAV inputs that usually come in a pattern. If there’s a sudden change in the pattern, our software flags the anomaly.”

Cyber tools can also help analysts monitor the chatter that moves over various Web channels. Computers can find links between various online avatars and aliases, sifting through millions of transmissions to find communication patterns that may show closeness with known bad actors.

“The software can look at an actor’s handles and see who they’re talking to,” Tran said. “There’s such a conveyor belt of data that it’s difficult for humans to know what’s important or what’s not. It’s not just the huge volume of daily data. Computers can look at communications over far longer time spans.”

Beyond the battlefield

Battlefield uses get a lot of attention, but big data techniques aren’t limited to ISR. Many projects focus on the support infrastructure. Big data techniques, for example, can help maintenance personnel reduce downtime and guide programmers to potential bugs.

One critical focal point is to reduce the amount of time that vehicles and other equipment spend in the shop. There’s a major effort to move to predictive diagnostics, which tell maintenance personnel when problems are imminent. Prognostics also maximize routine maintenance, monitoring parts so services are provided only when necessary instead of on scheduled times.

“When the hard disk drive comes off a V-22,* it has all the characteristics from the flight,” Raney said. “We can see the number one problem with V-22s and examine the sensor readings around the problem point. We can build a database that tells us which readings are indicators that a failure is imminent so repairs can be done when they’re necessary.”

Big data techniques are also being used to help DOD stay ahead in the technology race. One Defense Advanced Research Projects Agency program—Mining and Understanding Software Enclaves, or MUSE—aims to help programmers find open-source modules that can be plugged into their application packages. Software analysis techniques can help these programmers save time by determining whether the program will work and whether it has any potential problems.

“Military programmers are increasingly relying on open source. There’s a push to use it more for both front line and back office applications,” said Suresh Jagannathan, a program manager at DARPA. “Most of it is good. As long as the errors are small, we can mask out the errors. We look at similarities with known good Java and C programs that behave in a similar way, comparing them to a new open-source program. If the new program has an anomaly, we can track it to see if the anomaly is a bug or a vulnerability.”

These techniques are particularly useful in the large, complex programs used to manage many different levels of manpower and equipment. In these multifaceted programs, it’s far harder to find problems than in smaller applications.

“The real benefits arise in programs that are complex with mission-critical characteristics,” Jagannathan said. “You can save a tremendous amount of time and cost by synthesizing open source programs. When there are hundreds of thousands of lines of code, big data techniques can spot the anomalies and detect bugs.”

The human factor

While technology is the central element in big data analytics, humans obviously play an important role. All computer systems must be designed so people can gain benefits without investing a lot of time. System designers must focus on the ways users interact with systems and those in charge of users must be sure that maximizing the results provided by big data systems is part of the military culture.

If big data schemes are to work, all parties need to understand that their cooperation is a critical factor. Even with DOD’s focus on building the Joint Information Environment and sharing information, many commanders want to hold onto their data, citing a range of reasons. But computer systems tasked with finding links within large data sets will be more efficient and effective if they have as much relevant data as possible. Commanders and other personnel will have to be trained to pass their data on for further analysis.

“Getting access to data is the first part of the problem,” Raney said. “We also have to change the mindset from ‘need to know’ to ‘need to share.’ That’s a cultural change that may take a while.”

It’s also important to remember that the point of big data is to limit the data that reaches a user’s eyes. Analytics systems will send analysts alerts when the computer finds something that needs further attention. But the programmers who set up the big data parameters will have to be sure they’re not overloading operators with false positives.

“If we send too many alerts, users will not trust the system,” Buck said. “We need to work closely with the operators to avoid that. We want to present data to analysts and let them make the decisions. We also need to provide warfighters with a way to select the information they want for their requirements. It’s not a one-size-fits-all solution.”

There’s another spot where big data and humans come together—when it might actually be better for humans, rather than computer programs, to do search and analysis. This approach can be particularly effective when it’s difficult to precisely categorize search parameters. For example, in the event of a large societal conflict, thousands of military personnel could be tasked with examining border crossings—a tactic that is part of a research program for natural disasters to help the Navy, which is often a first responder.

“We’re looking at crowdsourcing,” Buck said. “Instead of using a computer for analysis, thousands of people look at images for something like a large disaster, when there might be thousands of square kilometers of damage. People can pick out problems. We don’t need to train the computer to find what we want.”

*Correction: An earlier version of this story incorrectly quoted Raney as saying "B-22."