4 strategies for dealing with the coming data tsunami

For the military, big data and analytics tools have become essential for sorting through the mountain of intelligence being collected from a rapidly multiplying number of human and machine-based sources. Deriving actionable insights from raw data, then rapidly disseminating that information to key users across the intelligence community and the military is a challenge that lies at the heart of ensuring continuous and reliable situational awareness.

This challenge for military organizations is becoming profoundly more acute as the number of sensors – and volume of data collected by these sensors – explodes. In June, the Office of Naval Research (ONR) issued a call for proposals to advance a robust Naval Big Data Ecosystem, with a major goal of the effort to, “…significantly improve the Naval community's real-time ability to bring together information from National systems and sensors with information from Naval combat and weapon systems and sensors.”

More recently, the Defense Advanced Research Projects Agency’s Multifunction RF (MFRF) program stated on its aim to address degraded visibility for combat helicopter pilots by developing multifunction sensor technology for pilots to operate safely in degraded and zero-visibility conditions; avoid collisions; and improve target detection, identification and engagement. 

Advances in hardware and wireless network technologies have led to low-cost, low-power, multifunctional sensor devices that can be affixed to military vehicles, satellites and soldiers, in turn making the big data challenge for the Defense Department even bigger. At the same time, the proliferation of unmanned vehicles – and the sensors mounted onto them – has generated a tidal wave of battlefield data that military leaders must ensure is properly digested, analyzed and acted upon.

Technology strategies to address the data tsunami

Done well, big data analytics uncovers valuable nuggets of useful intelligence hidden inside mountains of structured, unstructured, real-time and legacy data assets. The trick to finding these knowledge fragments is to use the analytical tools that reveal the greatest amount of useful information in the least amount of time. Successfully preparing for and managing the sensor data tsunami can come down to conquering four key challenges.

1. Developing a comprehensive approach to using big data

With an eye towards growing data volume, military organizations have focused on what used to be called PED – processing, exploitation and dissemination. Air Force doctrine, for its part, now expands that term to PC-PAD – planning and direction, collection, processing and exploitation, analysis and production and dissemination. The Air Force’s PC-PAD approach is a step in the right doctrinal direction, but all the services – indeed all relevant interagency operational participants – must combine and centralize their data for this approach to be truly effective.

2. Getting the right information to decision-makers

XMG analyst Jacky Garrido states that organizations must use analytics “to avoid getting buried under the humongous amount of information they generate through various outlets.” As intelligence, surveillance and reconnaissance (ISR) data is gathered and integrated, for example, it must also be both mapped out and separated from insignificant or unnecessary “noise” that is irrelevant to decision-making.

3. Finding effective ways of turning “big data” into “big insights” 

Data in and of itself doesn’t provide decision-makers with the kind of insights they need to do their jobs effectively or make insights into future conditions. The right analytics tools and people – talented analysts or “data scientists” – are needed to help operational leaders make sense of the volumes of data that are pouring into their organizations. Data visualization tools are particularly effective in ISR analysis as they place data into context both geographically and temporally. 

4. Finding and training the right analysts: Big data skills are in short supply