Why Agentic AI Breaks Every Existing Governance Framework
The Pre-Computation Fallacy: Five frameworks. One assumption. The math breaks all of them.
Executive Summary
Five governance frameworks. Five different organizations. One shared assumption.
The EU AI Act requires providers to document intended purpose before deployment. NIST requires reliability assessment under conditions of expected use. The OWASP Top 10 for Agentic Applications prescribes per-tool restriction profiles. Singapore’s Model Governance Framework for Agentic AI requires bounding risks and limiting scope of impact at the planning stage. ForHumanity CORE AAA mandates pre-deployment specification of scope, nature, context, and purpose.
Each framework was built independently. Each arrived at the same foundational requirement: describe what the system will do before the system does it.
Agentic AI systems are architecturally designed to determine their own execution paths at runtime. They select tools. They chain actions. They compose workflows nobody anticipated at assessment time. The system described in the compliance documentation is not the system running in production. It stopped being that system the moment the agent made its first autonomous decision.
This is the Pre-Computation Fallacy — the structural assumption embedded in every major governance framework that behavior is describable before the system operates. It is not a gap to close with better documentation. It is a mathematical constraint that documentation cannot overcome. And every downstream governance artifact — the risk assessment, the conformity assessment, the human oversight design, the incident response plan — inherits the flaw the moment the upstream assumption fails.
The organizations that recognize this will build the governance architecture that survives enforcement. The ones that do not will discover during their first regulatory inquiry that their compliance documentation describes a system that no longer exists.
The Comfortable Lie
Here is what the market wants to believe:
If you document the system’s behavior thoroughly enough before deployment, you have governed the system.
This is the foundational promise of every compliance program, every management system, every conformity assessment on the market. Document the intended purpose. Assess risks within those boundaries. Certify against the documented baseline. Monitor for deviation.
The promise works for deterministic systems. A medical device performs the same function on Tuesday that it performed on Monday. A financial calculation engine produces outputs from a bounded set of inputs. The documentation describes the system. The system matches the documentation. The auditor verifies the match.
Agentic AI breaks the match.
The system does not perform the same function on Tuesday that it performed on Monday. It performs a function it composed for itself based on the tools it selected, the data it retrieved, and the action chains it assembled — none of which were specified at deployment. The documentation describes Monday’s system. Tuesday’s system built itself at runtime.
The compliance market has not absorbed this. The governance industry is selling documentation discipline for systems that outrun documentation by design.
The Math
An agent has access to ten authorized tools. It can chain actions up to ten steps deep.
Ten tools. Ten chaining steps. Ten billion possible workflows.
That is not a metaphor. It is combinatorics. N actions across D chaining steps produce N^D possible compositions. The outcome space grows exponentially with every additional tool or chaining depth the agent is permitted. Even with constraints that reduce the practical space, the growth remains exponential.
Three tools across three steps: 27 workflows. Documentable.
Five tools across five steps: 3,125 workflows. Difficult.
Ten tools across ten steps: 10,000,000,000 workflows. Impossible.
No quality management system documents ten billion workflows. No risk assessment bounds them. No monitoring system watches all of them. No human reviewer evaluates a meaningful fraction.
The governance specification requires the organization to describe what the system does. The math does not allow it.
This is the constraint. It is not a resourcing problem. It is not a tooling gap. It is not a maturity deficit. It is an exponential function operating against a linear governance requirement. More documentation does not help. Better documentation does not help. The space the documentation needs to cover grows faster than any organization can write.
Every framework that requires pre-deployment behavioral description runs into this wall. None of them name it.
Five Frameworks, One Assumption
Each of the five major governance frameworks requires the provider or deployer to describe what the system will do before the system operates. Each breaks on the same structural constraint.
EU AI Act
The regulation requires providers of high-risk AI systems to document the system’s intended purpose, its foreseeable conditions of use, and its capabilities and limitations — before the system is placed on the market. The conformity assessment evaluates this documentation. The regulatory architecture assumes that the system described in the documentation is the system that will operate.
Where it breaks: an agent’s intended purpose changes at runtime. Tool selection, action chaining, and workflow composition produce operational purposes nobody declared and nobody documented. When the agent composes a workflow that enters an Annex III domain — creditworthiness assessment, employment screening, law enforcement support — the classification assessment filed at deployment no longer describes the system in production. The documentation was accurate when it was written. The system it describes no longer exists.
What it costs: behavior that exceeds the documented scope constitutes a potential substantial modification. The deployer may assume provider obligations for a high-risk system nobody registered.
NIST AI Risk Management Framework
NIST requires reliability assessment under conditions of expected use. MAP requires documenting the full scope of agent tools and autonomy boundaries. MEASURE requires metrics for trustworthiness characteristics. The framework assumes the conditions of use are knowable in advance.
Where it breaks: the conditions of use for an agentic system are not knowable in advance. The agent determines its own conditions of use at runtime by composing tool calls into workflows. The expected-use specification describes a subset of what the system can do. The system operates across the full compositional space. The reliability assessment evaluated conditions the system has already departed from.
OWASP Top 10 for Agentic Applications
OWASP prescribes controls including per-tool restriction profiles — capability allowlists, schema validation, rate limits. Each tool gets its own security boundary. The approach assumes that securing individual tools secures the workflow.
Where it breaks: tool-level security does not equal workflow-level security. An agent with authorized access to a customer database, a communication API, and a scheduling tool can compose a workflow that sends unsolicited messages to customers using data retrieved from the database, scheduled at times calculated to maximize response rates. Each tool call passes validation individually. The composed workflow was never assessed. The security controls see components. The regulatory obligation covers the composition.
This is not a criticism of OWASP’s work — the Top 10 is the strongest practitioner-facing taxonomy available. It is a structural observation about where per-tool controls reach their architectural limit.
Singapore Model Governance Framework for Agentic AI
Singapore’s MGF requires organizations to assess and bound risks upfront and limit the scope of impact at the planning stage. It is the only framework globally that explicitly addresses agentic AI governance. It recommends constraining agents to documented operational boundaries and testing both individual and multi-agent interactions.
Where it breaks: bounding risks at the planning stage assumes the risks are enumerable at the planning stage. For compositional systems, they are not. The agent’s impact scope changes with every workflow it composes. The boundary documented at planning time describes the system as planned. The system in production assembles its own scope.
Singapore’s framework comes closest to acknowledging this — its emphasis on continuous monitoring and kill-switch capability reflects an awareness that planning-stage controls alone are insufficient. The gap is in the assumption that planning-stage risk bounding can produce the reference frame continuous monitoring needs to monitor against.
ForHumanity CORE AAA
ForHumanity requires pre-deployment specification of scope, nature, context, and purpose. The Algorithmic Risk Committee establishes monitoring standards and the boundaries within which agent behavior is assessed. The multi-agent governance scheme explicitly states that combining systems beyond the provider’s pre-determinations makes the deployer a provider of a multi-agent system.
Where it breaks: the pre-determination requirement assumes the behavioral space can be pre-determined. For agents that compose workflows at runtime, it cannot. The provider pre-determined scope A. The deployer combined it with systems B and C. The agent composed a workflow across all three that nobody pre-determined. The deployer has become a provider of a system whose behavior nobody specified.
ForHumanity’s multi-agent governance provision is the most direct acknowledgment of this structural problem in any certification framework. The gap is upstream: the pre-determination requirement that feeds the multi-agent trigger cannot be satisfied for compositional systems.
The Downstream Cascade
When the upstream assumption fails, every downstream governance artifact inherits the failure.
The risk assessment evaluated risks within the documented behavioral space. The agent operates outside it. The risks it encounters were never assessed.
The conformity assessment certified the system described in the documentation. The system in production is a different system. The certification describes Monday’s system. Tuesday’s system was composed at runtime.
The human oversight design was built to oversee the documented workflows. The agent composes workflows the oversight function was never designed to evaluate. The reviewer does not recognize the workflow as outside scope because nobody defined where scope ends.
The incident response plan was written for anticipated failure modes. The agent produces failure modes nobody anticipated because the compositional space is too large to enumerate. The first time the organization encounters the failure is during the incident.
Each downstream artifact was built correctly against the upstream specification. The upstream specification does not describe the system that is running.
This is the cascade. It does not require malice. It does not require negligence. It requires only that an autonomous system did what autonomous systems are designed to do — determined its own behavior at runtime.
The Convergence
The Pre-Computation Fallacy is not only a compliance problem. It is simultaneously a security problem.
The OWASP Top 10 for Agentic Applications catalogs how agents fail under adversarial pressure — goal hijacking, tool misuse, identity abuse, memory poisoning. The EU AI Act specifies what providers and deployers must prevent — unauthorized behavioral drift, inadequate oversight, undocumented modifications. The overlap exists because both frameworks target the same architectural property: behavioral predictability.
When an agent’s behavior exceeds the pre-computed space, the security team sees an anomaly. The compliance team sees a potential substantial modification. The incident report and the regulatory case file describe the same facts.
The security team does not read the regulation. The compliance team does not read OWASP. They are both looking at the same agent. Neither has the full picture.
The Pre-Computation Fallacy is the structural reason they are looking at the same event through different lenses. The assumption that behavior is describable before runtime is the assumption that makes security monitoring possible and the assumption that makes compliance documentation valid. When it fails, it fails for both teams simultaneously.
One event. Two frameworks. Zero shared vocabulary.
The organizations that survive enforcement will be the ones that recognized the convergence before the incident forced them to do so.
The Response
The Pre-Computation Fallacy does not have a documentation solution. No amount of pre-deployment specification solves an exponential constraint.
It has an architectural response.
The operational envelope does not attempt to describe the full compositional space. It defines the subset of behaviors the organization actually assessed — the bounded region where the risk assessment, the conformity assessment, the human oversight design, and the incident response plan remain valid. Everything inside the envelope was evaluated. Everything outside it is unknown territory.
The governance mechanism is detection, not prediction. A tripwire inside the boundary. When the agent’s behavior crosses that boundary, what happens next is a human decision — not an agent decision and not an automated remediation.
Four questions define the envelope. A detection architecture makes it operational. A response protocol converts boundary crossings into governance events. A documentation framework makes the whole thing defensible.
That methodology was published in full in the piece Governing What Your Agent Does Next of this newsletter. The architecture is available. The question is whether organizations build it before the first enforcement inquiry reveals that their compliance documentation describes a system that no longer exists.
The Verdict
Five frameworks. Five different organizations. Five different regulatory traditions. One assumption.
System behavior can be described before the system operates.
For every system that preceded agentic AI, the assumption held well enough. For agentic AI, it fails — structurally, mathematically, and operationally.
The governance frameworks are not wrong to require documentation. They are wrong to assume documentation can capture a compositional space that grows exponentially with every tool the agent is authorized to use.
The Pre-Computation Fallacy is the name for the gap between what governance requires and what the architecture allows. Every organization deploying agentic AI is operating inside that gap. The ones that name it can build around it. The ones that do not will discover it during enforcement.
Name it before the regulator does.
Zero-Day Dawn publishes enforcement intelligence on agentic AI governance every Sunday at 4:00 PM EET. If you build, deploy, or govern AI agents — the gap between what you assume and what survives enforcement is widening every week. Paid subscribers get the full map.
Regulatory Disclaimer
This article provides educational analysis of the EU Artificial Intelligence Act (Regulation (EU) 2024/1689) and related governance frameworks. Nothing in this article constitutes legal advice, regulatory interpretation, or compliance certification. Organizations should consult qualified legal counsel specializing in EU AI Act compliance before making classification determinations or deployment decisions. Quantum Coherence LLC does not provide legal advice or regulatory compliance determinations.
Sources
EU AI Act (Regulation 2024/1689), Articles 3(23), 6, 9, 11, 14, 15, 25, 43, Annex IV. NIST AI Risk Management Framework 1.0 (January 2023). OWASP Top 10 for Agentic Applications (December 2025). Singapore Model Governance Framework for Agentic AI (IMDA, January 2026). ForHumanity CORE AAA Multi-Agent Governance v1.5 (2026).
True, most jurisdictions are still operating on static governance, assuming behaviour can be defined upfront. That breaks quickly in agentic systems.
I do see MAS and IMDA moving toward more adaptive governance and real-time guardrails, early days, though.
The harder part will be implementation. We’ve seen this before with frameworks like Basel III - strong in intent, but uneven in how institutions actually translate them into execution.
Great article Violeta, this is precisely some of the challenges faced in other domains where previously complex deterministic systems are now integrating AI/ML. I would like to draw parallels to a similar situation in the Autonomous Vehicle domain , where the scenarios the vehicle can encounter can not be fully bounded and the various permutation and combination of situations is uncontainable. To provide guidance for it , we have the SOTIF(Safety of the intended function) standard - ISO 21448. It helps in identifying the 'tripwire' and expanding the perimeter of it.
Of course there is not a whole lot of overlap to the AI Agent world, but some fundamental concepts can be borrowed.