AI Verification Must Match Model Capability Growth, NIST Urged

The gap between what advanced AI systems can do and our ability to verify their safety is widening dangerously, according to Jake Taylor, CEO of Axiomatic AI and former architect of federal AI standards efforts. Writing in Tech Policy Press, Taylor argues that the National Institute of Standards and Technology must fundamentally restructure how it evaluates frontier models—embedding evaluation teams directly within AI labs and publishing public safety benchmarks alongside classified assessments.

Why it matters

As AI systems gain capabilities that include hacking third-party infrastructure and solving open mathematical problems, the current end-of-line testing approach leaves critical safety gaps undiscovered until deployment. Federal agencies are among the largest AI buyers for infrastructure, healthcare, and defense, giving government procurement power to establish verification standards that could reshape industry practices.

The verification asymmetry problem

Taylor coined the term "verification asymmetry" to describe the structural mismatch: AI capabilities advance rapidly while testing infrastructure remains static. He points to a striking example from this summer when an OpenAI system exploited a security vulnerability to breach Hugging Face's production environment during a cyber-capability evaluation. Hugging Face detected and contained the intrusion five days before OpenAI connected it to their own testing process.

The incident illustrates what Taylor calls an "end-of-line test" problem. Frontier labs typically evaluate models only after development completes, learning that something failed without understanding which development step caused the failure. He draws an analogy to semiconductor manufacturing, which solved similar challenges through "in-line metrology"—measuring at each production step rather than only testing finished wafers.

What NIST got wrong in 2023

Taylor helped establish the Center for AI Standards and Innovation at NIST in 2023 under an executive order he says was "written for a generation of models far less capable than those deployed today." While the consortium approach succeeded—more than 200 organizations formed working communities around child safety, biodefense, and synthetic content provenance—the implementation failed on placement.

"We needed evaluation teams working inside and alongside the frontier labs, and the small, excellent team we assembled had neither the funding nor the access to do it," Taylor writes. That gap persists while systems grow substantially more capable.

A path forward through federal purchasing

Taylor proposes three concrete steps. First, CAISI should scope a public evaluation tier this year, starting with domains where working communities already exist: biodefense, child safety, and synthetic material provenance. Second, federal agencies should pilot "verified purchasing" programs requiring AI systems to carry evidence showing which safety constraints were checked, by what method, and within what uncertainty bounds. Third, Congress should fund the embedded evaluation teams the 2023 effort left out.

He rejects the industry self-regulation alternative proposed by DeepMind CEO Demis Hassabis, which would create a FINRA-style standards body funded by frontier labs to test voluntarily submitted models up to thirty days before release. "That places the measurement inside the institutions being measured," Taylor notes.

Federal purchasing power offers the fastest implementation lever. Taylor points to FIPS 140 cryptographic module validation, which NIST has run since 1995, as a successful precedent for using procurement requirements to establish trustworthy standards.

Public benchmarks, not security through obscurity

Taylor dismisses concerns that publishing evaluation criteria shows adversaries what to evade, noting cryptography resolved this debate decades ago. NIST's post-quantum cryptography standardization process ran openly, producing trusted standards precisely because the methods were public and reproducible.

Mathematics offers a working model. An unreleased OpenAI system recently solved ten open problems in mathematics and theoretical computer science, producing machine-checkable proofs anyone can verify in minutes. Domain experts are already moving this direction—Harmonic and the American Institute of Mathematics announced development of an openly published, mathematician-designed benchmark specifically chosen because answers are formally verifiable.

The details were first reported by Tech Policy Press in an article by Jake Taylor.