An Honest Automation Dashboard Separates Output, Outcome, and Risk
By Equipo Quantum Developers
Summarize:
Operating thesis
“Ten thousand cases processed” proves activity, not value. Volume can rise while rework grows, margin falls, or exceptions become invisible. The honest dashboard has a strict thesis: no output metric should be presented as ROI without an associated outcome, visible denominator, verification coverage, and a risk signal capable of stopping automation.
Google SRE recommends selecting indicators that measure the service users care about and connecting objectives to a response in Service Level Objectives. For agents, that discipline prevents easy telemetry from standing in for useful performance. The dashboard should take a committee from what happened, to what changed, to what exposure was created.
The three layers
Layer 1: output
This describes flow, not benefit:
- cases received, eligible, processed, and excluded;
- share in recommendation, approval, and execution;
- cycle time by state;
- queue, age, and exceptions by reason;
- technical and human cost per case;
- availability and dependency failures.
This layer diagnoses capacity. Never label it “impact.” Automation rate needs a denominator: eligible cases, not every received case. It also needs quality. Automating a case that later reopens can increase output and worsen operations.
Layer 2: outcome
This shows the change the process intends to produce:
- confirmed time to resolution;
- error, reopen, or rework;
- protected margin, recovery, or realized cost;
- compliance with an operating window;
- capacity actually reassigned;
- incremental outcome against a comparison when appropriate.
The UK Treasury Magenta Book distinguishes process, impact, and value evaluation and emphasizes the counterfactual for attributing effects. The dashboard need not teach the entire method, but it should label an outcome as observed, comparative, or causally attributed. “After the agent” does not mean “because of the agent.”
Layer 3: risk
This exposes possible harm and control effectiveness:
- decisions without an observable outcome;
- material actions without valid approval;
- evidence coverage and age;
- errors by severity rather than average alone;
- failure concentration by population;
- actions blocked by policy;
- time to detect, escalate, and contain;
- accepted residual risk and owner.
NIST calls for context-relevant metrics, tracking of emerging risk, control evaluation, and documented uncertainty in the AI RMF Core. An incident counter at zero does not prove safety when outcomes are unobserved or severity is unclassified.
Artifact: the one-page scorecard
| Executive question | Primary measure | Denominator or coverage | Decision rule |
|---|---|---|---|
| Is work flowing? | cases closed within the window | expired and unexpired eligible cases | reduce admission when the queue ages |
| Did the outcome change? | outcome difference from baseline or comparison | population with observable outcome | do not monetize without adequate coverage |
| Was value realized? | savings, used capacity, or avoided risk | attributable benefit and full cost | revisit scope when unit cost rises |
| Is it controlled? | decisions inside policy | decisions subject to the policy | pause on a material breach |
| Can we explain it? | complete evidence packets | closed cases | block closure without evidence |
| Who responds? | exceptions assigned within the window | exceptions needing a person | add capacity or reduce autonomy |
Thresholds belong to the enterprise. The table offers no benchmarks. Its purpose is to stop a favorable card from appearing without the condition that could invalidate it.
Anti-vanity rules
Never show a numerator alone. “Five errors” lacks volume, severity, and population.
Do not average irreversible and trivial outcomes. One payment error does not vanish among hundreds of correct classifications.
Do not monetize theoretical capacity. Released time is capacity until finance proves savings or reassignment.
Do not mix states. Extraction, proposal, approval, and execution represent different automation.
Do not hide exclusions. An agent can look accurate by routing every difficult case away.
Do not report quality without coverage. Quality is known only where outcomes are observed and reviewed.
Do not use color without action. A red card needs an owner and response.
Do not change the definition silently. Formula, source, and version travel with the metric.
An illustrative example
Consider a quote queue. Output shows more documents generated and shorter preparation time. Outcome, however, shows no margin improvement against a comparable cohort. Risk reveals that discount exceptions wait longer and some cost evidence is stale.
No numbers are needed for the illustrative decision: autonomy does not expand. The team keeps document generation, repairs cost freshness and approval capacity, then measures margin again. An activity dashboard would declare success; the three-layer view finds the new bottleneck.
Data design before visual design
Every card needs:
- name and definition;
- data owner and response owner;
- population, denominator, and exclusions;
- source, frequency, and delay;
- formula and version;
- coverage and uncertainty;
- segmentation by consequence;
- threshold and action.
Join layers through the business-object and run identifiers. In Quantum Automation Center, states, timelines, artifacts, logs, and operational and financial analytics can connect output to evidence. Systems of record and process owners still validate the economic outcome.
Cadence and audience
Operations needs queue and exception data while work happens. Product needs trends and segments to change scope. Risk needs severity, control, and evidence. Finance needs realization and cost. The executive committee receives few measures but can drill down to population and case.
Do not create four truths. Share definitions and tailor the view. When the monthly report uses a formula different from the operating dashboard, the discussion shifts from decisions to disputed numbers.
The strongest counterargument
Three layers may overwhelm executives and weaken a story that needs a few memorable measures. Outcomes and risks also arrive later than output, making the dashboard look incomplete or unfair to a new pilot.
Answer with hierarchy and maturity. Show one lead measure per layer and state “not yet observable” instead of filling the space with activity. Allow drill-down. Explicitly missing evidence is more useful than a precise number about the wrong thing.
When not to use this approach
Do not build an ROI dashboard before defining the outcome, eligible population, and accountable decision-maker. During discovery, a learning and cost log may be enough. Do not force monetization onto an obligation or control whose purpose is not savings.
Use the scorecard once the agent processes repeated cases and a committee must expand, constrain, or retire it. The honest dashboard does not seek an all-green screen. It ensures a favorable decision cannot hide the operating cost or risk that made it possible.
Sources
Article topics
