The US Intelligence Community Is Caught in a Collector’s Trap

“Everything will be in its blind volumes. Everything, but for a rational line or just notion, there will be millions of nonsensical cacophonies. Everything, but generations of men can pass without finding among the vertiginous shelves a single tolerable page.” — Jorge Luis Borges

Imagine spending your entire life wandering an endless labyrinth, searching for meaning and finding little but garbled nonsense. Argentine poet Jorge Luis Borges described such a life in his eerily prescient short story “The Library of Babel.” Borges’ tale is an allegory about humanity’s pursuit of scientific knowledge. But it also serves as a compelling metaphor for the challenge that has entrapped modern intelligence services.

The Library’s size was indefinite, perhaps infinite. It was said to contain every book ever written—indeed, that ever could be written—because it held in some far corner or another a tome with every possible combination of written characters. Most of these were worthless since they were full of nothing but gibberish. Some, though, contained snippets of tantalizing coherence, others even flashes of brilliance. It was thought by the library’s inhabitants that their labyrinth must contain books with the solution to every possible problem, just waiting to be discovered. The odds of finding these were so infinitesimally slim, however, that it drove many of them to suicide.

Like one of Borges’ lonely librarians, an intelligence officer might spend her entire career in search of some exquisite piece of information: for example, the proverbial needle in the haystack that warns of an imminent terrorist attack. And like the Library itself, the global datasphere in which they search is practically limitless. 

This haystack has in fact been doubling in size every two years for the last decade, and that growth is likely to accelerate over the next. Yet while the datasphere grows geometrically, the mechanisms intelligence services use to make sense of it—spies, listening posts, and satellites—can only be added arithmetically. The gap between information to collect and information that is actually collected keeps growing larger, and can never be closed.

But not for lack of trying. For decades, the U.S. intelligence community has added to its expansive data-collection enterprise. Today, it costs the taxpayer around $80 billion a year. National Intelligence University researcher Josh Kerbel calls this the community’s “classified collection business model.” It is premised upon the idea that information and intelligence are essentially synonymous, and posits that the time and treasure spent gathering and sorting it all is justified because it leads to better policy choices—what the community calls “decision advantage.”

The record of American foreign policy failures in my lifetime alone, however, suggests that this model is flawed. Policymakers today have access to more timely, more accurate, and more voluminous information than at any time in history, yet we’ve nonetheless witnessed a parade of foreign policy missteps and embarrassing self-owns that have sapped the nation’s capacity, credibility, and resolve. 

It is tempting to be comforted by periodic coups d’éclat such as the collection feats that led to the deaths of al-Qaeda founder Osama bin Laden and Islamic State leader Abu Bakr al-Baghdadi. But remember: for every impressive tactical collection coup, there are equally, if not more impressive strategic collection catastrophes. These include the failures to anticipate Russia’s snap invasion of Ukraine in 2014, its Syrian incursion in 2015, and its interference in our own elections in 2016.

The intelligence community has been seduced by the characteristically American conceit that it will one day be able to encompass all the world’s information, a delusion I call the collector’s trap. This technophilic hubris has led the community to double down on collection each time it has failed, confident the answer must be more collection. 

But even if it were possible to gather every bit of information relevant to national security (it isn’t), it wouldn’t serve policymakers as well as you might think. We tend to find the things we look for and are surprised when the things we are not looking for find us instead. 

What makes the collector’s trap so insidious is not only its elusive goal but also the fact that the more information we do gather, the more confused we become. Human susceptibility to cognitive errors such as availability bias and the observer-expectancy effect means that with a virtually limitless amount of information already at our fingertips, certainty about practically anything has only decreased.

Caught by Our Own Snare 

In the past, information was scarce. Useful information about foreign competitors was scarcer still and thus valuable. It made sense, then, to steal as much of it as possible. Early states employed rudimentary intelligence networks to gather what information they could in the first place, and to protect their own from being gathered by their opponents in the second.

Carl von Clausewitz, the doyen of modern war, held this sort of intelligence in low regard. He defined it as “every sort of information about the enemy and his country,” and dismissed most of it as false, worse than useless. It was suitable, he wrote, only for promoting fear and uncertainty, because soldiers were more apt to believe bad news than good. 

But while that is undoubtedly so, I will be so bold to say that Clausewitz was wrong. Or, at least, his scorn was misplaced. It turns out that he himself provided a profoundly succinct justification for intelligence elsewhere, he just didn’t recognize it as such. More on that in a moment.

By the time the modern intelligence community was established in the immediate aftermath of World War II, the scale of that challenge had grown so vast that Sherman Kent, one of the community’s founding fathers, observed that “…to be able to deliver [intelligence] in the fashion apparently expected… would demand a research staff large enough to codify and keep up-to-date virtually the sum-total of universal knowledge.”

Contemporaries criticized Kent as having believed events were like a “tape all printed up inside a machine,” with the job of intelligence officers being to tell the planners what it said—an unfair exaggeration. But Kent did believe that intelligence problems were like puzzles that could be solved if only the right piece of information could be found. If estimates turned out to be wrong, the remedy was simply to collect more information.

As the Iron Curtain was drawn across Europe, the budding intelligence community attempted to do exactly that. 

The classified collection model’s architects built a sprawling intelligence-gathering armature whose ambitions, in time, would brush the limits of what was possible. But the model’s fundamental logic stayed intact throughout the Cold War because the most useful information—war plans, missile designs, and the like—remained in the hands of a relatively small number of elites.

Today, the situation is reversed. In sharp contrast to Kent’s era, useful information is now quite literally everywhere. Rather than belonging to few, it can come from practically anyone, anywhere on earth. And due to the growing complexity of international affairs, more information does not equate to more accurate forecasts. 

Accordingly, both the efficiency and the efficacy of the collection model have been called into question, for reasons I hope are becoming clear. The first is that while for most of its existence the intelligence community enjoyed a monopoly on the useful information provided by the systems it owned and operated, the value of that information has now plummeted as the market for it has become saturated. The second is because of the cognitive errors that afflict us all — the two mentioned already, plus many more — our problem now is not too little information, but too much. 

The intelligence community has over-invested in technical collection platforms at the expense of the people who give the information those systems collect context. Today’s consumers of intelligence are drowning in data, but thirsting for insight.

A generation ago, analog photography giant Eastman-Kodak dominated its field, as it had for well over a century. Its executives knew the future was digital, in fact, they had invented the world’s first digital camera way back in 1975. Their problem was not ignorance; it was that they were over-invested in the status quo. Kodak profited from every aspect of analog photography: they produced the cameras themselves, of course, but they also made the flashcubes, the film, the chemicals used to develop the film, even paper finished photos were printed on. The digital future looked very disruptive to Kodak executives, and they chose to simply ignore it. 

Today, the intelligence community owns the collection platforms, as well as the exploitation and processing centers, the communications channels, and even the administrative infrastructure that controls access to classified information. 

The intelligence community is not a business, nor is it motivated by profit. But it should apply the lessons learned by other institutions adapting to change, and its efforts should profit the nation. The time to begin thinking about a pivot is long overdue. The intelligence community must chart a bold new model suited to the information-rich reality of our digital era, and finally, break free from the collector’s trap. 

The Truth Shall Make You Free 

The nation’s intelligence services should conform to the nation’s needs. In the past, when useful information was scarcer, the need was to find pieces of it wherever they hid and use them to build a coherent picture. What today’s America needs most, in contrast, is help making sense of a shapeless, increasingly discordant world where information is abundant but truth is in short supply.

Today, intelligence can no longer be synonymous with information, or even with secrecy—the very notion of which is dying. Soon, there will be no more hiding in an increasingly transparent world that is always monitoring, always tracking, and always listening. When information is cheap and easily accessible, what is valuable is discernment and curation. 

Let’s get back to Clausewitz. He said critical analysis is “tracing effects to their causes,” that is, illuminating the connections between things to determine “…which among the countless concatenations of events are the essential ones.” This is a solid value proposition for intelligence in the information age. He further warned against treating separately what is more accurately viewed as a gestalt, and reminded us that individual actions, “however small their cause, must influence all subsequent military operations and modify their outcome to some degree, however slight.” In other words: It’s all connected, man.

Related: After 9/11, US Intelligence Reinvented Itself. It's Time to Do It Again

Related: Spies Like AI: The Future of Artificial Intelligence for the US Intelligence Community

Related: Border Agents Can Now Get Classified Intelligence Information. Experts Call That Dangerous.

My argument is not that we should stop collecting secrets, only that we should place less emphasis on doing so and realign limited resources accordingly. Knowing what foreign leaders are saying behind closed doors can provide valuable context, and every commander wants to know as much as possible about their enemy before meeting in battle. Hard intelligence collection problems will, of course, endure. 

But because we can’t possibly collect everything, or even everything we think may be relevant, we must put far more emphasis on cultivating anticipation and foresight. We must become comfortable with uncertainty rather than trying to eliminate it. We must expect surprise, and grow more resilient, adaptive organizational structures and policies better suited to endure and incorporate the lessons learned from them. 

At base, intelligence leaders must remind themselves that they are not in the business of collecting and protecting information, but of delivering insight and facilitating understanding so that better decisions can be made to advance national interests.

Borges may have anticipated the digital age when he conceived the Library in 1939, a scant few years before Claude Shannon realized the potential of binary code and set us on the path that saw it manifest a half-century later. But like Babel, Shannon’s quantification of information—the bit—makes no distinction between sense and nonsense. Intelligence is concerned with what Shannon almost dismissively referred to as semantic information, that is, information that has been imbued with meaning. And only people can do that.