The Pilot Purgatory Problem
The Department of Defense and the broader federal government have invested heavily in AI over the past five years. Executive orders, national strategies, and service-level AI implementation plans have generated real momentum. Hundreds of AI projects have been initiated across the defense and intelligence communities. Yet a consistent pattern has emerged: promising AI capabilities that perform well in controlled pilot environments fail to transition to operational use.
This phenomenon — sometimes called pilot purgatory — is not primarily a technology problem. The AI models work. The demonstrations are impressive. But the organizational, procedural, and institutional barriers to moving from a successful pilot to a deployed operational capability prove insurmountable for a disturbingly high percentage of projects.
The Government Accountability Office, the RAND Corporation, and the DoD's own Chief Digital and AI Office (CDAO) have all documented this challenge. The causes are consistent: unclear pathways from experimentation to programs of record, insufficient attention to data infrastructure, lack of sustained funding beyond initial research dollars, and the absence of operational users in the design process from the beginning.
Data: The Unglamorous Foundation
The most common reason AI projects stall is not model performance. It is data. Organizations embark on AI initiatives without first ensuring that the data required to train, validate, and operate AI systems is accessible, clean, labeled, and governed. The AI team builds an impressive model on curated demonstration data, then discovers that the operational data is in a different format, lives in a system they cannot access, contains errors that corrupt model performance, or is classified at a level that prevents the development team from working with it.
Data infrastructure is not exciting. It does not generate the kind of attention that a successful AI demonstration does. But it is the foundation on which every AI capability depends. Organizations that invest in data engineering — data pipelines, quality assurance, labeling workflows, metadata management, and governance — before they invest in AI models consistently achieve better outcomes than those that build models first and discover data problems later.
The federal data strategy and the DoD Data Strategy both emphasize this point. But strategy documents do not build data pipelines. The gap between strategy and implementation is where most organizations struggle, particularly when data engineering competes for the same funding as more visible AI development work.
The Acquisition Mismatch
Federal acquisition processes were designed for hardware programs with predictable development timelines, fixed requirements, and defined milestones. AI development follows a fundamentally different pattern: iterative experimentation, continuous learning, evolving requirements as the technology matures, and performance that improves with more data and operational feedback. These two paradigms are deeply incompatible.
A traditional requirements document that specifies AI system performance in advance — "the system shall classify targets with 95% accuracy" — assumes a level of predictability that AI development does not offer. Performance depends on the operational data environment, which may differ significantly from the training environment. Requirements should specify operational outcomes and evaluation frameworks rather than fixed performance metrics.
Similarly, the funding model for AI programs must accommodate continuous development and improvement. An AI system that is deployed and never updated will degrade as the operational environment changes and adversaries adapt. The DevSecOps model — continuous integration, continuous delivery, and continuous monitoring — is the right paradigm for AI, but it requires sustained operations and maintenance funding that traditional program structures often fail to provide.
Lessons for the Next Wave
The organizations that have successfully transitioned AI from pilot to operations share several common characteristics. First, they involve operational users from the earliest stages of development, ensuring that the AI capability addresses a real operational need rather than a technology team's assumption about what would be useful. Second, they invest in data infrastructure before model development, treating data as a strategic asset rather than an afterthought.
Third, they establish clear transition pathways before the pilot begins — identifying the program of record or operational organization that will own the capability, the funding that will sustain it, and the authority to operate in the target environment. Fourth, they adopt iterative development approaches that deliver incremental capability rather than attempting to build the final system in a single development phase.
The federal government's AI ambitions are appropriate to the scale of the challenges it faces. But ambition without disciplined execution produces demonstrations, not capabilities. The second wave of government AI adoption must learn from the failures of the first, investing in the unglamorous work of data infrastructure, acquisition reform, and organizational change that makes operational AI possible.
