Meet the Man Reinventing CIA for the Big Data Era

The CIA will stand up its new Directorate for Digital Innovation on Thursday. It's the first directorate the agency has added since 1963 and the biggest change to America’s key spy service since before the moon landing. The new office will look beyond the spycraft of today to the very big question of how to turn the vast amounts of data that the agency collects into useful insight for analysts, agents, the agency, and the nation. The goal is to turn chatter and daily digital exhaust into a window into the future.

Defense One visited CIA headquarters in Langley, Va., for an exclusive interview with CIA Deputy Director Andrew Hallman, picked by CIA director John Brennan to lead the new office.

To encounter Hallman is to meet a man who is, in appearance and mannerism, the living embodiment of a more stately era of spying. He’s poised, articulate, thin, looks at home in a suit with suspenders and his favorite show is “Mad Men.” (He seems both more adult and more a Boy Scout than Don Draper.) Though he has the bearing of a Cold War operative, Hallman is also deeply in touch with the promise, potential, and hype surrounding big data and the interconnected era.

“The days of attending a cocktail party and writing up your notes are over,” Hallman told us. “A digital world challenges the way we work in a clandestine world. We have to come up with new ways to operate in a much more-connected environment and still be clandestine,” he said. Read that to mean: it’s hard to maintain a cover (or “legend” as the CIA calls them) in an era where everyone has a television studio in his or her pocket. The digital era presents a sea change potential for intelligence collection. That’s true for adversaries as well.

Hallman sees the new directorate solving three big problems for CIA. The first is helping agents and operatives hone their hacking and sleuthing skills. “The way we help people use digital and cyber techniques, this will raise it to a new level,” he said. Part of that is developing intelligence about those places where the United States can’t or won’t put boots on the ground, or at least, as many boots as the U.S. would like. “Yes, you can develop standoff access, depending on the target,” he said. “ISIL social media is a rich source of info for us ... They’re the most aggressive of actors in using social media, one of the most prolific. It’s elevated the importance of it as a source.”

The second problem: improving the CIA’s data management, or what Hallman calls “governance.” That sounds like fixing the filing system but Hallman says that better data governance will help CIA answer questions about precisely what it’s doing and what it isn’t doing. ”As taxpayers frame what we do, our ability to govern our data, our ability to respond to the public and to Congress is key. This directorate will help us balance those interests,” he said, referring to the balance between security and privacy. “We can be more assuring to Congress and the public in terms of how to govern that data.”

The third and greatest challenge Hallman is looking to solve through the directorate, taking the vast volumes of digital intelligence that CIA receives from around the world and transforming it into a moving, credible picture of the future. Intelligence, in this context, becomes almost a super power.

“We have the ability to do more sense-making to provide for analysts a real ability to forecast,” said Hallman, quickly adding that forecasting, arriving at a better understanding of multiple probabilities, is fundamentally different from prediction. “We are gaining the ability to anticipate the conditions of change to determine if they are anomalies or areas to focus on. This directorate will bring focus to that anticipatory intelligence capability,” he says. “It’s often in the aggregate that we get an impression.” That’s particularly true of social media, which provides of course not only intelligence for specific operations--targeting this or that ISIS leader in Syria--but also taking the temperature of an entire population. It’s sentiment analysis on a massive, population-wide scale. “If you have time, you can develop insight into patterns,” with social media data, he says.

What does that mean? In describing how the directorate will provide anticipatory intelligence, Hallman highlighted a 2011 program sponsored by the Intelligence Advanced Research Projects Agency, or IARPA, called Open Source Indicators, or OSI. The goal was to “develop methods for continuous, automated analysis of publicly available data in order to anticipate and/or detect significant societal events, such as political crises, humanitarian crises, mass violence, riots, mass migrations, disease outbreaks, economic instability, resource shortages, and responses to natural disaster,” according to the IARPA Web site. In other words, the agency wanted to take massive amounts of publically available data from news reports, social networks and every other available open information source and paint a picture of some event that might occur of national security significance, like a revolution or an assassination, perhaps.

The program achieved some notable success. In 2012, a team of Virginia Tech researchers affiliated with OSI effectively predicted two events on the basis of open source data from social networks and elsewhere. One was that Mexico’s election of Enrique Peña Nieto to the presidency would be met with protests. Sure enough, at 11:15 p.m. on July 1, the student group Yo Soy 132 began marching in protest against election results declaring Nieto the winner. The marches set off weeks of arrests and police actions. The Virginia Tech team also correctly forecast a series of protests around the impeachment of Paraguay’s President Fernando Lugo. Sure enough, on June 29, 2012, 2,500 people effectively closed the bridge linking to Paraguay Ciudad del Este to the Brazilian town of Foz de Iguaçu. The team’s methods likely would have predicted other flare-ups had they not made the specific decision to limit themselves to Central and South America.

When asked if that was the sort of thing Hallman wanted to be able to do everywhere, he answered “yes.” He called OSI, “an important feature in our resources and development…a cutting edge program.” To have real relevance, the directorate will need to be able to predict change through sentiment or other means before it’s obvious to everyone else, even the participants. Lead time is the difference between magic and a bad press conference.

Better anticipatory analyses in the hands of agents and analysts will require a lot of new software to allow agents to directly tap into the agency’s largest data sets and pull out insights directly relevant to them and their current operations. The directorate will create new application program interfaces, or APIs, toward that end and will also pioneer entirely new mathematical approaches to data analysis and extrapolation. Hallman already has plans to hire a chief data officer to spearhead the development of advanced analysis techniques, to “manage the data science cadre and provide standards for development APIs that allow various applications to talk to each other,” he said. (It’s safe to say it will be the sexiest job in data science outside of Facebook.)

The result, he hopes, will be agents and analysts developing a slightly more prophetic understanding of the operation in front of them, the mission at hand, and doing so across a variety of devices. No it’s not a crystal ball, but it could help in decision-making in a way that enables to agency to focus on the right suspects better and faster.

Everything about that--writing software, creating new APIs, and big data analysis--represents an evolution beyond the role that the CIA traditionally has played, the collection of human intelligence on foreign subjects for national security. It elevates a new role that involves the collection of signals intelligence. But signals intelligence collection and analysis has long been the job NSA. Hallman says that there will “always be a distinction” between the two agencies. But today’s human targets produce a lot of digital data, and as targets become more digital, the two missions are merging somewhat, becoming more integrated.

“We can use human-enabled targeting of signals intelligence in a way that’s more operationally relevant. That means a greater ability to provide a human-enabled way to help the NSA target, to be very selective in how we apply our exquisite forms of collection,” he said.

The notion of a United States intelligence agency, one that’s caught a lot of recent criticism from both lawmakers and from the public, taking on big data quickly become a controversial one. When asked if he anticipated public suspicion toward the new directorate, Hallman answered: “There might be. But the directorate will have a strong public profile...” It will have to, since it will have to work with more corporate partners than the agency does traditionally, which forces a certain level of transparency.

It goes back to the governance challenge. Narrower intelligence collection and analysis will--at least in theory--result in less need to use large amounts data from more people, a key consideration in the post-meta data era. When intelligence collection becomes more focused, fewer people get caught up in the data dragnet. That’s a net privacy win, says Hallman. “It’s a benefit,” he says. “The public debate about signals intelligence is reaffirming. It reaffirms our need to apply our intelligence collection to where it has the most value.”

Of course, big data has been the beneficiary of a lot of hype and hasn’t met with everyone’s expectations. It’s proven effective for predicting consumer behavior and elections, bad for predicting the outcome of sporting events and the stock market. In other words, in domains where two sides have a lot of options for out maneuvering each other, where the game has more variables, it fails. Will it be useful for the CIA?

Perhaps, for Hallman the metrics for success start with changing the way the agency uses technology and data. “We will be successful when we know that digital and cyber is central to all we do. There’s a fast and rapid increase in technological ability,” in the hands of both friends and adversaries. “We need to be able to keep pace.”

Keep pace, and hopefully increase it.