DOD falls short in measuring SBIR effectiveness, GAO says

The Defense Department needs to develop better tools to measure technology transition for its Small Business Innovation Research (SBIR) program, says a recently released General Accountability Office report.

While the program has been successful in transitioning some technologies into acquisitions programs or fielded systems, the overall efficiency of the program remains unknown because DOD does not collect comprehensive and reliable data on transitions, the report found.

Instead, most of the information on transitioning technologies is collected on an impromptu basis.

“The military departments collect information on selected transition ‘success stories’ on a somewhat ad hoc basis from SBIR program officials, acquisition program officials, prime contractors, or directly from small businesses,” according to the report.

The services also use two data systems—Company Commercialization Reports and the Federal Procurement Data System—to monitor program-wide transition results, GAO said. But both systems have significant gaps in coverage and data reliability concerns—neither was designed to collect detailed data on projects that did not transition, fielded systems or acquisition programs.

In an analysis of these data systems, the most common issue GAO found is that they fail to capture all commercialization data. Each system has a tendency to be limited to certain types of contracts. For instance, CCR sources only get data on small businesses that are seeking additional SBIR awards, resulting in limited data on participants that aren’t seeking new awards

The 2012 National Defense Authorization Act required DOD to begin reporting on SBIR transition information, but the department is still figuring out how to deal with several challenges including:

An unstandardized definition of technology transition. A challenge found in 2013 was that the military branches had differing definitions of technology transition, making measurement difficult. GAO has recommended that DOD establish a common definition to meet yearly reporting requirements.

Lengthy transition times. SBIR officials have found that long periods of time—sometimes several years—between SBIR project completion and the actual transition to DOD make it difficult to link the technology back to SBIR.

Lack of resources. Officials have also stated that limited resources for administrative activities result in increased difficulty in following up on transition outcomes. Meanwhile, military users and weapon systems acquisition programs often do not spend resources on monitoring SBIR projects.

Developing better measurement techniques would allow DOD to determine the effectiveness of the SBIR program and if the program is using valid approaches to develop and transition technologies, the report said. Measuring the impact of these programs could allow DOD to develop new management procedures.

“In an environment of declining budgets, it is important that information on technology transition outcomes for SBIR projects be improved for DOD to identify the extent to which the program is supporting military users and determine whether existing monitoring and transition efforts are working effectively,” the report said.