An approved demo clears only one of seven production gates
By Equipo Quantum Developers
Summarize:
A demonstration can complete its happy path and still be unfit for a single real decision. The input was selected, the operator knows the script, and someone can repair any surprise just outside the frame. An approved demo therefore clears only one of seven gates: production needs evidence for value, data, behavior, authority, security, operations, and launch.
The goal is not more paperwork. It is to change the question from “does it work?” to “under which conditions will we accept operating it, and who can stop it?” The NIST AI Risk Management Framework Core treats governance, mapping, measurement, and management as continuing functions and calls for documented scope, human oversight, and component risks. A screen recording cannot answer those operational questions.
The production acceptance pack
The pack needs one index. Each gate points to versioned artifacts, an owner, and an exit decision. “Not applicable” is valid only when someone explains why and accepts the residual risk.
| Gate | Acceptance question | Minimum exit evidence |
|---|---|---|
| 1. Outcome and owner | Which decision or task improves, and who owns the result? | Scope, baseline, owner, and success rule |
| 2. Data and rights | Is input valid, timely, and authorized for this use? | Contract, quality, lineage, retention, and access |
| 3. Behavior | Does it meet its contract outside the demo script? | Evaluations, adverse cases, limits, and version |
| 4. Authority | What may it inform, recommend, execute, or escalate? | Decision matrix, approvals, and prohibitions |
| 5. Integration and security | Can it create duplicate, improper, or irreversible effects? | Permissions, idempotency, secrets, reversal, and test |
| 6. Operations | Who observes, supports, and recovers the service? | Metrics, alerts, runbooks, support, and handoff |
| 7. Launch | How will it be exposed, stopped, and evaluated on live work? | Staged plan, launch owner, rollback, and review |
The gates are dependent, not a checklist to complete at the end. A data discovery can change behavior evaluation. An authority limit can require a different integration. A recovery scenario can reveal that the business outcome cannot tolerate the human decision delay.
Gate 1: outcome, cohort, and accountability
Name the business object and the specific decision. “Help finance” is vague; “classify reconciliation exceptions for analyst decision” establishes a testable boundary. Document the included cohort, a comparable baseline, and conditions that invalidate the comparison. Do not invent a savings percentage before observing outcomes.
Exit requires a business owner able to accept quality and consequences, not merely a budget sponsor. If nobody can explain what outcome justifies operating risk and cost, the system remains a technology demonstration.
Gate 2: the data and rights contract
List each source, identity, schema, observation time, freshness expectation, permitted purpose, access path, and rejection rule. Test absence, duplication, format changes, and out-of-scope records. A screenshot of clean data proves presentation, not resilience.
Exit includes a retention and evidence decision. Keeping everything “just in case” may conflict with the authorized purpose; keeping only the final answer makes a decision impossible to explain. The pack states what is retained, where, and who may inspect it.
Gate 3: the behavior contract
Specify output format, facts that require evidence, deterministic constraints, prohibited actions, and the expected response to uncertainty. Evaluate representative and adverse cases, separating adjustment examples from acceptance examples. Record model, instructions, tools, and policy because “the same agent” is not a useful identity after any of them changes.
The gate does not demand perfection. It demands known limits and a rule that routes uncertain cases to review. An acceptable average cannot compensate for a small error class with severe consequences.
Gate 4: decision rights
Build a verb matrix: read, summarize, recommend, draft, send, modify, commit funds, cancel, and approve. For each verb, state scope, condition, approver, and required evidence. Human oversight is real only when the person has the context, time, and authority to stop the action.
Exit must demonstrate that the agent cannot jump from an authorized verb to a final effect. A prompt that says “ask permission” is not a technical permission or policy gate.
Gate 5: integration, security, and reversibility
Review least privilege, secret isolation, parameter validation, tool limits, idempotency, and observed remote state. Test the case where a response disappears after a write. Design compensation for reversible effects and prior approval for irreversible ones.
Include external dependencies and degraded behavior. When a critical tool is unavailable, the agent should stop, queue, or downgrade to read-only under visible policy—not improvise another path.
Gate 6: operability
The AWS Operational Readiness Review guidance recommends validating architecture, process, event management, and release quality before launch, then repeating the review during the lifecycle. For an agent, supply dashboards, actionable alerts, failure classes, playbooks, contacts, handoffs, and support capacity.
Exercise containment and recovery for each relevant failure mode. Exit is not “the alert arrived.” It is “an owner interpreted evidence, stopped the appropriate effect, and verified recovery.”
Gate 7: launch and exit control
The Google SRE chapter on reliable launches describes launch coordination and checklists that transfer reliability knowledge into a product. For an agent, name one person empowered to advance, pause, or revert, then define staged exposure through observable cohorts.
The plan should use shadow operation where useful, human assistance before autonomy, and a rollback rule tied to signals rather than sentiment. It also needs a review date: accepting production does not mean accepting it forever.
The evidence manifest
The pack index can remain compact when it is precise:
- agent identity and version;
- business object, cohort, and outcome owner;
- data contract and evaluation;
- behavior report and known limits;
- decision and permission matrix;
- architecture, dependencies, and reversal;
- dashboards, alerts, runbooks, and support;
- launch decision, exceptions, and signatories.
Every artifact has a version, owner, date, and gate relationship. A presentation that copies conclusions without linking evidence is not the pack.
In Quantum Automation Center, the manifest can connect catalog identity, version, execution status, timeline, artifacts, logs, permissions, and approvals. The Automation Center documentation describes governance and observability surfaces that help preserve this relationship. The gate owners—not the interface—still make the acceptance decision.
The counterargument: gates can slow learning
Giving the same weight to an internal assistant and an agent that commits money creates control theater. Scale the evidence to impact, reversibility, and scope. A gate may accept a lightweight artifact or an explicit not-applicable decision, but it should not disappear silently. That preserves speed while exposing the assumption the organization accepts.
When not to use the complete pack
Do not apply it to an isolated exploration with no users, real data, persistence, or external actions. An objective, data boundaries, and a learning log are enough there. Do not duplicate certified controls for a deterministic tool merely to fill boxes; link the existing evidence.
Use the pack once a pilot will touch real decisions. The final question is not whether the demo impressed. It is whether all seven accountable functions can point to evidence, accept residual risk, and stop the agent when that evidence is no longer valid.
Sources
Article topics
