The ACLU Used Amazon’s Facial Recognition and it Labelled Congress Members as Criminals

AI researchers have repeatedly found that machine learning software picks up biases in the data from which it learns. This bias has been demonstrated in text, images, and audio, and now the American Civil Liberties Union (ACLU) is bringing this message to US lawmakers.

A test by the civil-rights organization trained Amazon Rekognition, the company’s machine learning tool specifically meant for facial recognition, to match faces with 25,000 publicly-available mugshots, and then had the algorithm try to match mugshots with images of members of the US Congress, according to a blog post from ACLU lawyer Jacob Snow. Not only did the algorithm falsely match 28 members of Congress to criminal mug shots, but the false matches were more common for congresspeople of color.

The ACLU is using this demonstration to call for a moratorium on police use of facial-recognition technology, claiming that it’s both too biased and not accurate enough to be used effectively.

“If law enforcement is using Amazon Rekognition, it’s not hard to imagine a police officer getting a ‘match’ indicating that a person has a previous concealed-weapon arrest, biasing the officer before an encounter even begins,” Snow wrote, “Or an individual getting a knock on the door from law enforcement, and being questioned or having their home searched, based on a false identification.”

Related: You Can Protest, But You Can’t Hide From Government Facial Recognition (For Much Longer)

Related: Chinese Police Are Wearing Sunglasses That Can Recognize Faces

Related: US Army Figures Out How To Do Facial Recognition in the Dark

A representative for Amazon Web Services told Buzzfeed News that the ACLU’s approach could possibly have been improved by changing the confidence level the algorithm must have before confirming a match between two pictures—basically how sure the algorithm is that aspects of the image overlap. The representative said that AI apps should have different confidence levels based on their importance—for example, a food-recommendation algorithm and one that’s used in law enforcement should be held to very different standards. But Snow said that he used the default settings in Rekognition.

The ACLU did not immediately respond to a request for comment.

In reaction to the ACLU’s work, representatives Jimmy Gomez and John Lewis have sent a letter to Amazon CEO Jeff Bezos asking to meet and talk about the facial recognition technology, according to Buzzfeed News.

“We want to express our deep concerns about your facial recognition technology … we are concerned that a [facial] match will exacerbate existing biases and strip people of their Constitutionally protected rights,” the letter says.

Update: This story has been updated to include representative Gomez and Lewis’ letter.