How the Intelligence Community Can Get Better at Open Source Intel

During his House Appropriations Committee testimony on May 27, Chairman of the Joint Chiefs of Staff Gen. Mark Milley presciently stated that a country must master ubiquitous intelligence-surveillance-reconnaissance to win the next major war. Indeed, in an era in which the United States can no longer rely on economic or military overmatch to guarantee its security, we need a more intimate understanding of an adversary’s goals, intentions, capabilities, and actions to safeguard our national interests. The chairman’s challenge begs important questions. How do we ensure that our civilian and military leaders enjoy decision advantage in an increasingly contested environment? Channeling the mindset of Sun Tzu, how can we win before the first shot is fired—or prevail without any shots at all? 

Senior leaders consistently assert that the key to decision advantage in an ISR construct is timely acquisition and analysis of the best information. Historically, the analysis produced to meet intelligence requirements was based predominantly on data acquired by government collectors and government technologies. Unlike open data sources, intelligence officers can task sensitive intelligence sources and methods to target the specific people, places and events that drive the intelligence needs of our policymakers and commanders. 

But the world has changed. In today’s digital age, people and organizations carry out activities using technology that projects data about their background, actions, and preferences onto public platforms. Simultaneously, private companies have built business models around co-opting the two pursuits that animate most intelligence work—identifying and predicting personal behaviors. The combined result of these phenomena is a data ecosystem that couples extraordinary quantities of information with sophisticated processing tools to produce stunningly diverse insights—including insights relevant to the hardest intelligence problems. 

The ubiquity and accessibility of this public data disrupts the assumed superiority of the government’s proprietary intelligence sources of methods—a reality the national security community has been slow to recognize and accept. The open-source intelligence, or OSINT, derived from the vast pools of publicly/commercially available information, or PAI/CAI, in the public domain will not replace traditional intelligence—but at the very least it can enrich and enhance this data, and for many intelligence requirements it may be a better, safer and cheaper option of first resort. OSINT can also provide overwatch for areas and topics that fall outside the zone of active IC coverage. 

Unfortunately, we are poorly postured to realize these benefits because technology and industry have far outpaced the law, policy, and governance required to optimize the use of OSINT in the national security community. Current national security frameworks did not anticipate the explosion of internet-based PAI, the increased demand for public-private sector collaboration, or the wide availability of certain data aggregated and sold by private companies that the government could not legally collect on its own. We need an expedited policy response that generates the laws, policies, resources, and tradecraft required for the IC to confidently bring the full force of OSINT to bear on the serious threats we face, while ensuring that the acquisition and exploitation of PAI/CAI protects the privacy of U.S. persons and adheres to the ethics and practices of the intelligence profession. 

One of the most formidable obstacles to this end-state is cultural. Within the IC the production of OSINT is not regarded as a unique intelligence discipline but as research incident to all-source analysis or as a media production service. This approach was reasonable when open data sources consisted primarily of media and academic output. Now, however, given the volume and diversity of internet-derived PAI/CAI, OSINT more closely resembles SIGINT than library research. In the IC, the SIGINT discipline is a strong center of gravity—managing a diverse array of unique data types, each with its own unique collection, discovery, processing, exploitation, tradecraft, sharing and oversight processes—governed and championed by a dedicated agency. 

OSINT, on the other hand, remains a distributed activity that functions more like a collection of cottage industries. While OSINT has pockets of excellence, intelligence community OSINT production is largely initiative based, minimally integrated, and has little in the way of common guidance, standards, and tradecraft. The conspicuous absence of common standards, community coordination, and enterprise governance make PAI/CAI more difficult to use in all-source assessment than classified information produced by the IC’s more established disciplines. SIGINT, GEOINT and HUMINT reporting, for example, is well understood by analysts, has established tradecraft, and is supported by a deep bench of specialists who help interpret and evaluate the reporting. In the world of open source, the lack of common tradecraft and reach-back support to discipline specialists makes evaluating the accuracy and validity of PAI/CAI challenging, and all-source analysts may be understandably reluctant to use that reporting—substantially diminishing the value of a significant resource. The intelligence community must make OSINT a true intelligence discipline on par with the traditional functional disciplines, replete with leadership and authority that enables the OSINT enterprise to govern itself and establish a brand that instills faith and trust in open source information. Only then will OSINT have the advocacy, commitment, and structure to move from a cottage industry to the core discipline it must become.

A second significant obstacle is resources. OSINT activity is not funded at a level nearly commensurate with its value, though congressional intelligence oversight committees have pressed repeatedly for greater use of OSINT. Foreseeably, the absence of a dedicated OSINT advocate and champion results in severe underinvestment in open source information at a time that our national security posture demands a Manhattan Project mentality to harvesting the immense intelligence value of open source data. Instead, the nascent OSINT initiatives in each intelligence community element compete within their organization’s budgetary Thunderdomes, leading to predictably anemic outcomes. 

Finally, we need to update the antiquated data handling rules and policies designed to regulate older technologies and mitigate different risks. The astronomical volume of internet derived information, innovative machine analytics, and the proliferation of private commercial data providers present a unique opportunity for the intelligence community to achieve a scope of geographic and topical coverage unimaginable ten years ago. However, the lack of enterprise-wide legal and policy guidance impedes the widespread use of PAI/CAI, and increases the risk of inappropriate activity when the scant existing guidance is interpreted inconsistently by agencies across the community. The question of what constitutes an inherently governmental collection function, for example, directly influences the formation of partnerships between the intelligence community and commercial data providers. The private sector is witnessing a proliferation of commercial companies which not only provide unique data, but also bundle that data with analytic tools that enable new and striking insights. Maximizing the utility of OSINT requires the open source community to partner with industry in much the same way that the GEOINT community currently partners with commercial imagery providers. We need anticipatory policy that enables the IC to safely, effectively and ethically team with the private sector to evolve critical open source collection and analytic capabilities.

Operationalizing OSINT is crucial to the success of the intelligence community and to the security of our nation. Creating a foreign-intelligence OSINT capability for an era of peer adversaries and heightened global uncertainty will be challenging. Adjusting structure, policy, and resourcing will require a concerted effort across Congress, the Office of the Director of National Intelligence, and the Department of Defense. The intelligence community must act boldly to shift OSINT into a true intelligence discipline. Policy and oversight must keep pace with efforts to nurture the public-private partnerships that enable government to leverage the rapid technical advances occurring in the commercial and NGO sector. Resource programming must support these efforts and align budget expenditures to realize the tremendous and growing value of OSINT to the intelligence mission. 

The adversaries our nation faces are very real, very capable, and very dangerous. We can no longer afford complacency. Now is the time for audacious action to realize the Chairman’s vision of ubiquitous ISR, and aggressive exploitation of PAI/CAI will be the linchpin. The competition for decision advantage starts well before the first shot is fired and spans all the national instruments of power: diplomatic, information, military, and economic. Technology and data available in the public domain could close critical knowledge gaps now. Government must match and embrace these capabilities with commitment, policy, and resources. Failure to do so risks disastrous consequences. Our adversaries will capitalize on these opportunities and so must we – we are already behind. 

The views expressed are the authors’ and do not imply endorsement by the Office of the Director of National Intelligence or any other U.S. government agency. 

Bob Ashley is a former director of the Defense Intelligence Agency and now an adjunct senior fellow at the Center for a New American Security. 

Neil Wiley is former Principal Executive in the Office of the Director of National Intelligence and a former Chair of the National Intelligence Council.