DIU seeks evaluation infrastructure solutions for AI capabilities

DIU seeks evaluation infrastructure solutions for AI capabilities

The Defense Innovation Unit is looking for an artificial intelligence evaluation "harness" to test mission-specific benchmarks in support of robust, repeatable, vendor-agnostic assessments. “As artificial intelligence (AI) capabilities evolve at an extraordinary pace, the government requires evaluation infrastructure that can keep pace by continuously assessing new models against mission-specific benchmarks as they are released,” DIU said in the…

Pentagon, IC want industry to provide an ‘evaluation harness’ to standardize testing of AI systems

The Defense Department and Intelligence Community are on the hunt for an “evaluation harness” to test vendors’ AI technologies for government use. The Pentagon’s Defense Innovation Unit, headquartered in Silicon Valley, released a solicitation Wednesday for the effort, dubbed “MYSTIC DEPOT,” which will be pursued via a commercial solutions opening contracting mechanism. The release comes as Defense Secretary Pete Hegseth and Pentagon CTO Emil Mi…