DARPA trying to make autonomous systems more trustworthy

The Defense Advanced Research Projects Agency wants autonomous systems to be able to evaluate how well they're doing a specific task and explain that to their human partners.

In a broad agency announcement issued Feb. 19, DARPA outlined the Competency-Aware Machine Learning (CAML) program, which aims to transform autonomous systems from tools into trusted partners by virtue of the systems' ability to evaluate their effectiveness and communicate that information to humans.

With CAML, machines will be able to match their behavior to human expectations and allow their operators to quickly understand how they are operating in complex, changing, high-stakes environments.

Today's machine-learning trained systems can adapt their behaviors to circumstances similar to those they've been trained on, but they are unable to communicate how they plan to carry out the task, the adequacy of their training relative to the job or other factors that could affect their probability of success.  That means humans must help systems make choices -- a poor use of resources in a combat environment.

CAML plans to create a machine learning framework for object recognition, robotic navigation, action planning and decision-making that will significantly improve teaming capabilities between humans and autonomous systems.  

The 48-month program will address three areas:

Self-knowledge of experiences, in which machine learning systems capture and encode experiences first discovered during tasks so that they can remember previous operations.

Self-knowledge of task strategies, in which systems analyze task behaviors, summarize them into general patterns and identify task dependencies.

Competency-aware learning, which establishes a learning framework through which descriptions of task strategies and expected performance can be communicated through a human-understandable, competency statement.

Initial testing will be performed on proposers' machine learning platforms and applications of choice, but the systems will eventually be tested on Defense Department platforms using realistic vignettes that evaluate the accuracy of the machine’s communication of its competency, not the human's perception of the machine’s competency. Examples of potential platforms include autonomous ground resupply vehicle platforms, UAV intelligence, surveillance, reconnaissance platforms and mission planning systems.

Scenarios might include autonomous vehicles "following the leader" while monitoring environmental conditions for necessary changes or a human asking various image recognition systems which is best at identifying a specific type of subject, such as cars and trucks.  

Responses are due April 22. Read the full BAA here.