Image analysis research targets photo fakery

A media forensics initiative unveiled last fall to help U.S. intelligence analysts detect doctored images has moved to the next phase, with a multidisciplinary team of university researchers working to pull together a platform capable of scanning millions of images and videos posted online each day.

As part of a four-year, $4.4 million project funded by the Defense Advanced Research Projects Agency, investigators said they would leverage machine learning along with image and signal-processing techniques to help spot images that might have been Photoshop-ed.

As groups like the Islamic State, also known as ISIS, increasingly turn to social media to recruit new members, U.S. intelligence agencies are relying more on online images and video to monitor the activities of terror groups. Since readily available software tools can be used to manipulate visual media, DARPA's Media Forensics, or MediFor, initiative will underwrite research to, for example, develop better algorithms that can be used to spot fake images. The tools would then allow analysts to conduct forensic investigations to determine precisely how and why images were manipulated.

That capability could ultimately provide insights into the "digital lineage" of doctored images and video, a field known as "multimedia phylogeny."

A prime example of photo fakery occurred in 2008 when Iran released a provocative image of a missile test that turned out to be doctored. The photo was widely published in U.S. newspapers before the deception was discovered.

In rolling out the program last fall, program officials said MediFor would attempt to integrate machine learning and image analysis technologies into a forensic-based platform to “detect manipulations, provide analysts and decision makers with detailed information about the types of manipulations performed, how they were performed… in order to facilitate decisions regarding the intelligence value of the image [and] video.” 

Image research has been divided among several U.S. universities along with investigators in Brazil and Italy. The multidisciplinary team includes the University of Notre Dame, New York University, Purdue University and the University of Southern California.

"A key aspect of this project is its focus on gleaning useful information from massive troves of data by means of data-driven techniques instead of just developing small laboratory solutions for a handful of cases," Walter Scheirer, a principal investigator at Notre Dame, noted in a statement.

Tools already exist to scan Internet images, but not on the scale required by U.S. intelligence agencies. Researchers noted that such a capability would require specialized machine-learning platforms designed to automatically perform processes needed to verify the authenticity of millions of videos and images.

"You would like to be able to have a system that will take the images, perform a series of tests to see whether they are authentic and then produce a result," explained Edward Delp, director of Purdue's Video and Image Processing Laboratory. “Right now you have little pieces that perform different aspects of this task, but plugging them all together and integrating them into a single system is a real problem."

Hence, investigators will attempt to piece together a complete system capable of handling the massive volumes of visual media uploaded to the Internet each day. That will require deep-learning tools capable of churning through millions of images, detecting doctored pictures and producing a digital lineage that might shed light on the motivation of terror groups.

Purdue's piece of the project focuses on using tools like image analysis to determine whether media has been faked, what tools were used and what portions of an image or video were actually modified. "The biggest challenge is going to be the scalability, to go from a sort of theoretical academic tool to something that can actually be used," Delp said.