Big data poses big challenge for military intelligence

The number of sensors deployed on the ground or aloft on unmanned aerial vehicles (UAVs) and satellites continues to soar, as does the ability to keep this asset in place for longer periods. As they generate enormous volumes of data, Defense Department planners are struggling with techniques that ensure that the data remains useful for warfighters.

The volume of data being gathered for intelligence, surveillance and reconnaissance (ISR) is already huge. In Afghanistan, ISR acquisition systems gather more than 53 terabytes of data every day, noted Under Secretary of Defense for Intelligence Michael Vickers. At the start of the war in Afghanistan, a single terabyte was considered a large volume of data.

“There’s a gap between our growing ability to collect data and our limited ability to process that data,” Gen. Robert Kehler, commander of the U.S. Strategic Command, said in late 2011. “We are collecting 1,500 percent more data than we did five years ago. At the same time, our head count has barely risen.”

Throughout the defense industry, there’s a concerted effort to deal with so-called big data. Tight budgets have led many to determine that it will be simpler to resolve the problems with technology than to boost the staff of skilled analysts who examine the surging volumes of data that show no signs of abating.

Developers are coming up with techniques that automate the analysis process. Some systems can now highlight changes that may be significant. They also can search for similarities in specified images. For example, a system may see that a red pickup is near the site of explosions, prompting analysts to do further searches or direct the system to search for other images that include the red pickup.