Category: AI Strategy

When you walk into most enterprise AI conversations today, the discussion starts with the model. Is it smart enough? Is the demo impressive enough? Does the feature set cover enough ground? But talk to buyers in operational, customer-facing, or regulated work and the questions shift. What job does this system do? What is it allowed to change? Who reviews the output? What happens when it is wrong? What business metric improves?

Those questions point to something specific. The unit of adoption is the governed workflow, not the model. A productized AI offering is a bounded operating loop with a trigger, an input, an action boundary, a system of record, a human-control point, an exception path, an audit trail, and a measurable outcome. That gap, between a model that runs in a demo and a workflow that runs with controls, is the difference between a pilot and an offer.

Pick the job first, then the AI

When you look at a supply-chain workflow that starts with a driver check-call, the job is clear enough that you can define the AI’s role without much debate. A late arrival triggers a check. If the delay is real, the system can update the shipment record, reschedule a dock appointment, notify the right team, or escalate to a human. The process has a boundary. It has a handoff. It has a point where the system acts and a point where a person remains responsible. That is not a generic “AI for supply chain” story. That is a bounded exception path with rules.

The evidence points in the same direction. Recent research on enterprise AI adoption identifies workflow redesign, human validation rules, KPI tracking, and operating-model integration as the difference between usage and scale. McKinsey’s 2025 survey found broad AI usage but far less scaled adoption. High performers were more likely to redesign workflows and define when humans validate outputs. Stanford’s 2026 enterprise AI research found that in 42% of studied production deployments, the underlying foundation model was fully interchangeable. The advantage sat in the application and orchestration layer.

The same pattern shows up in insurance. Claims intake, first notice of loss, service triage, and document review are credible starting points because they are repetitive, costly, and measurable. They also carry enough structure to support controls. Many complex decisions still route to human approval. That is often what makes the offer acceptable, not a weakness of it.

The best starting point is rarely the largest possible use case. It is the highest-return workflow with the lowest acceptable risk profile. If the work is important but too open-ended, the AI becomes a liability. If the work is safe but trivial, it never earns budget. The sweet spot is where the buyer can see the pain, define the boundary, and accept the control model. That is why “AI for operations” fails as a pitch. It is too wide to govern and too vague to buy.

The model is rarely the real failure

When AI deployments fail, people blame the model, and most of the time that is the wrong diagnosis. An agent can optimize for speed, skip validation, and still create compliance failures, integration failures, customer confusion, or downstream rework. The model may have done exactly what it was instructed to do. The operating assumptions were wrong. The system was optimized for the wrong metric inside an underspecified process. A strong model inside a weak workflow does not create a deployable product. It creates a faster way to make the wrong decision.

The public examples make this plain. McDonald’s ended its IBM AI drive-thru test in 2024 after a narrow, high-volume workflow still proved brittle in the real world. Air Canada was held liable after its chatbot gave a customer incorrect bereavement-fare information, and the tribunal rejected the idea that the chatbot was separate from the company. Deloitte Australia agreed to partially refund a government report after apparent AI-generated errors, including fabricated or unsupported references, entered a high-trust deliverable. Those were control, verification, and accountability problems, not only model problems. The lesson is not that AI should stay in the lab. What it teaches you is that automation without a control envelope is a risk transfer, not an offer.

Governance is part of the product

For customer-facing, regulated, or action-taking AI, governance cannot be added after the pilot works. A governed AI workflow needs four things: authority, observability, intervention, and accountability.

Authority defines what the system can access and what it can do. Observability records what it saw, decided, changed, and escalated. Intervention gives humans clear points to review, approve, override, or stop the system. Accountability names the owner of the result when the system is wrong.

This is product architecture, not a policy slogan. Microsoft’s agent-governance guidance frames agents as systems that access data, take actions, and operate with delegated authority. That means enterprises need to identify agents, assign ownership, limit access, observe behavior, and stop unsafe agents. Runtime authorization matters because the real question is not only whether a model response is safe. The question is whether this specific action should execute now, under this identity, policy, approval state, data boundary, and business context.

The better enterprise workflows already reflect that logic. Morgan Stanley’s AI Debrief is a clean example. A client meeting happens. With consent, the system captures notes, surfaces action items, drafts a follow-up email, lets the advisor edit and send, and saves a note into Salesforce. Morgan Stanley also reported 98% adoption of its earlier AI assistant across financial-advisor teams. The workflow is trusted because the control points are visible: consent, advisor review, discretionary send, and CRM persistence.

Governance also needs one more distinction. Human review is not automatically meaningful review. A human approval gate can solve accountability without solving quality if the human only rubber-stamps the machine. Automation-bias research and health-insurance commentary both show that humans can over-rely on automated recommendations. The control model has to define what kind of review is required: a quick edit, a compliance check, a clinical judgment, an operational override, or a full decision by a licensed person.

“Human in the loop” is too vague. The buyer needs to know which human, at which point, with what authority, and with what evidence.

Who carries the cost of error

Regulation is not a uniform force across all AI markets. The pressure is strongest where the workflow affects consequential decisions, regulated rights, customer harm, or public-facing obligations.

California’s SB 1120 shows what workflow-level regulation looks like in practice. In health-insurance utilization review, AI may inform an adverse benefit determination, but a licensed clinician must review the decision. The EU AI Act applies a similar logic more broadly through risk-tiered obligations tied to specific use cases. Colorado’s AI Act pushes high-risk deployers toward risk management, impact assessment, and human appeal.

This is the procurement reality. In consequential workflows, the buyer will ask who reviewed the output, who can override it, what evidence remains, what happens when the model changes, and who carries the cost of error. That question has moved from theory into contract language, regulation, and litigation. Recent enterprise AI contract patterns include AI-specific addenda, data-training restrictions, model-change notice requirements, AI system registers, and human-review obligations in acceptance criteria. The point is not that every buyer has mastered AI liability. Error ownership has become an active procurement variable, and that matters more than legal sophistication.

Healthcare litigation around AI-assisted coverage decisions shows the risk pattern. A bounded workflow can still create liability if review, escalation, and accountability are weak. Existing laws can assign responsibility even before AI-specific law matures.

For vendors, this changes packaging. A productized AI offer needs to say what it does, where it stops, and who owns the downside. A demo cannot answer that. A workflow design can.

Offer-ready is more than workflow-specific

A narrow AI idea is not automatically a product. McDonald’s proved that a bounded, measurable, and high-volume workflow can still fail when the real-world environment is noisy and the edge cases are ugly. Klarna showed a different version of the same lesson. Its customer-service AI initially looked like a flagship narrow-workflow success, handling two-thirds of customer-service chats in its first month. Later reporting showed that the company had to soften the replacement narrative because some customers preferred humans and complex issues still needed human agents.

These workflows were not too broad. The real issue was that workflow specificity alone did not settle the operating model. A narrow AI idea becomes offer-ready only when the vendor can define the job, the inputs, the systems touched, the action boundary, the exception path, the human-control point, the audit record, the KPI, and the failure owner. Use this test before you call an AI idea an offer.

The minimum viable AI offering canvas

1. What exact workflow, decision, or handoff improves?
2. Who uses it, funds it, and owns the result?
3. What does the AI do: recommend, draft, classify, trigger, execute, or coordinate?
4. What systems does it read from, write to, or change?
5. Where must a human review, approve, edit, override, or intervene, and what makes that review substantive?
6. What controls are required: permissions, logs, audit trail, escalation, and revocation?
7. What baseline, target metric, and time to impact define success?
8. What is the cost of error, and who carries it?

The canvas is not asking whether the AI is impressive. It asks whether the offer is commercially legible, governable, measurable, and priced to absorb the risk it creates. If you cannot answer those questions cleanly, you do not have a product. You have a promising capability looking for a job.

The market rewards bounded operating loops

The strongest examples are products wrapped around a bounded job. C.H. Robinson’s missed LTL pickup workflow is the cleanest case. A pickup is missed. The system checks the situation, decides the next step, calls the carrier, and pushes the freight back into motion. The value is an exception-resolution loop tied to visible operating metrics, not a claim about broad intelligence. C.H. Robinson reported that 95% of checks were automated, more than 350 hours of manual work were saved per day, freight moved up to a day faster, and unnecessary return trips fell by 42%.

The reason this example works is not that logistics is uniquely suited to AI. It works because logistics exceptions already have a shape: detect the exception, contact the responsible party, decide the next action, update the system of record, and escalate when the path breaks. AI fits because the operating loop already exists.

FourKites and project44 frame the same idea from another angle. Their products focus on carrier follow-up, ETA-triggered appointment changes, delayed-shipment rescheduling, document collection, freight-audit exceptions, and carrier onboarding. The value sits after the exception fires, when the system has to coordinate across tools, rules, and people.

Claims intake and first notice of loss show the same pattern. They are constrained, repetitive, and costly. They have enough volume to justify automation and enough structure to support controls. The buyer can imagine the job before the pitch is over.

The product becomes legible when the buyer can answer four questions in less than a minute:

• What job does it do?
• Where does it stop?
• When does a human step in?
• What outcome improves?

If the vendor cannot answer those questions, the offer is not ready.

Horizontal platforms still matter, but value is vertical

The argument is not that buyers only buy vertical AI. That would be too simple.

Horizontal platforms still attract major enterprise spend. Some companies want broad access first so employees can experiment, standardize identity controls, connect internal data, and build use cases on top. JPMorganChase’s LLM Suite is a good counterexample to any claim that buyers only buy one workflow at a time. Large enterprises with strong security, model-risk, and engineering capacity can buy or build a governed platform first, then attach it to workflows later.

But platform access is not the same as scaled operating value. The commercial unit is still the bounded job, even when the technical substrate is horizontal. Enterprises may sanction AI horizontally, but they scale it vertically. They may buy a platform for access, experimentation, and shared infrastructure. They count value when that platform changes a bounded process with an owner, a control model, and a metric.

Even the commercially strong platforms tend to win inside work surfaces. Salesforce works through CRM processes. Microsoft Copilot works best in roles where the work, data, and collaboration layer already live inside the Microsoft stack. Those are platform businesses, but their adoption still becomes concrete through jobs: qualify a lead, summarize a meeting, draft a response, prepare an analysis, resolve a case, update a record.

Broad platforms create access. Governed workflows create accountable deployment. Both matter, but only one carries the commercial outcome.

Start with the work, not the model

If you are evaluating an AI idea, do not start with the model. Start with the work.

Ask whether the job is narrow enough to govern, valuable enough to fund, and stable enough to support a clear human-control model. Ask whether the data exists, whether the exception categories are understood, and whether the buyer already tracks the metric that should improve. Ask who owns the result when the system is right, and who carries the cost when it is wrong.

If those answers are clear, the idea may be offer-ready. If they are not, the model is not the problem. The offer is not ready.

Most teams keep trying to sell intelligence. The buyer is trying to buy control.

One of the most overlooked questions in an AI SaaS evaluation is not “what features are included.” It is “what happens when we need this vendor to stop?”

That question sounds defensive until you watch a buying team realize what the demo is really showing. A customer-service platform is no longer only summarizing tickets or drafting replies. It is checking order status, updating CRM fields, issuing refunds under a policy threshold, escalating exceptions, and reporting resolution metrics. The feature checklist looks good, the integration plan looks manageable, and the ROI model looks plausible. But the real decision sits underneath the checklist. The buyer is deciding which part of the operation to hand over. Improve. Coordinate. Execute. Quietly become responsible for.

In my experience, AI SaaS buyers need a new buying lens. They should evaluate vendors on two axes at the same time: the operational role the vendor plays and the amount of work the vendor absorbs. The first determines where value appears. The second determines how hard the vendor will be to govern, pause, replace, or exit. AI SaaS does not make old lock-in disappear. In some deployments, it extends lock-in into the operating model. That is workflow absorption.

The risk is the delegation, not the agent

Traditional SaaS buying treated the application as the unit of value. The buyer asked familiar questions: does the product have the features we need? Does it integrate with our stack? What does it cost per seat? Will users adopt it? Can we export our data if we leave? Those questions still matter, but they are incomplete when the vendor starts acting inside the workflow.

AI is pushing SaaS value away from manually used applications and toward workflow-level outcomes. Applications move behind the scenes. The workflow becomes the front door. Agents coordinate work, trigger actions, reconcile data, manage exceptions, and sometimes participate in decisions. A customer-service platform may no longer be just a better ticketing tool if its agents resolve requests across systems. A finance platform may no longer be just an expense dashboard if it reviews transactions, enforces policy, flags fraud, approves low-risk spend, and logs the decision. A healthcare revenue-cycle platform may no longer be just administrative software if it checks claim status, works payer queues, and updates the EHR workflow.

This is why “AI SaaS” is too broad a category to guide buying decisions. A dashboard, a routing layer, a task-performing agent, and an expert-assisted decision system create different kinds of value. They also create different kinds of dependency.

Absorption occurs when a vendor becomes responsible for a repeatable operational function, rather than merely supporting the people who perform it. The threshold is easy to miss. A customer-service AI starts by summarizing cases. The vendor improves visibility. Then it recommends the next-best action, drafts a response, and routes the case to the right queue. The vendor now coordinates work. Then it updates a CRM record, changes an order, issues a refund, or closes the case. The vendor executes work. Then leaders rely on its resolution metrics, exception queues, escalation rules, and performance dashboards to manage the function. The vendor is no longer just a tool inside the workflow. It is part of how the workflow runs.

Public examples point in this direction, with an important caveat: the evidence supports bounded execution more than full autonomy. Salesforce Agentforce materials and customer-service analysis describe agents that can handle order status, refunds, troubleshooting, appointment scheduling, case updates, and profile updates. Intercom’s Fin has moved its pricing language from resolutions toward outcomes, including workflows where the agent gathers context, executes configured procedures, and hands off when policy requires a person. ServiceNow-like workflow platforms increasingly sit above ERP, CRM, and ITSM systems, turning the AI layer into the place where work is assigned and measured. Banking and payments examples now include agents involved in re-underwriting documents, payment dispute resolution, and transaction execution.

Agents have not taken over whole enterprises. They have not. What is changing is that agents are beginning to execute bounded tasks inside live workflows. Once that happens, the buying decision changes. The buyer has moved from software adoption to operational delegation.

The vendor absorption grid

Buyers need a simple way to name what they are delegating. The Vendor Absorption Grid starts with the vendor’s role.

Then the grid asks how much work the vendor absorbs into its own operating model.

The grid is a diagnostic, not a maturity model. Every buyer should not race toward absorption.

A visibility vendor may help leaders see bottlenecks faster. The dependency is usually manageable. A coordination vendor may improve prioritization and handoffs, but it also starts to shape who gets work, what gets escalated, and which cases receive attention. An execution vendor may expand capacity, but it creates continuity risk when the system updates records, triggers payments, closes tickets, or approves work.

The scrutiny changes most when a vendor participates in judgment. If the agent recommends eligibility decisions, refund approvals, dispute handling, risk scoring, claims handling, or pricing changes, the buyer needs to know what evidence a human sees, when a human can intervene, and who is accountable when the outcome is disputed. That does not make absorption wrong. In some workflows, a deeply embedded vendor is the rational choice. The vendor may be more capable, more consistent, and more accountable than a fragmented stack of tools and internal handoffs. But the buyer should make that trade-off explicitly.

What changes from classic lock-in

Enterprise buyers already know lock-in. ERP, CRM, ITSM, cloud, billing, procurement systems, RPA, BPO, and managed-services providers have always shaped processes, created switching costs, and given vendors control. AI SaaS does not invent dependency.

What changes is the compression of software, workflow orchestration, decision support, and managed execution into the same vendor surface. Some vendors are beginning to demonstrate pieces of this capability in bounded workflows: interpreting intent, selecting tools, routing work, updating records, escalating exceptions, and measuring outcomes. That compression matters because the system of record becomes less visible to the user. ERP, CRM, ITSM, billing, and support systems may remain underneath. But the AI workflow layer becomes the place where work is assigned, executed, reviewed, and measured.

Classic SaaS lock-in often centered on data, contracts, user adoption, and configuration. AI-era dependency can also span models, orchestration frameworks, runtime environments, prompts, policies, workflow configurations, agent behavior, action history, escalation patterns, and human operating routines.

Connection is not portability. A protocol may help one agent call another tool. It does not automatically make agent memory portable, preserve workflow configuration, transfer escalation habits, recreate behavioral calibration, or tell a new vendor why the old system routed a sensitive case a certain way.

The practical test is simple, even if the answer is difficult: could another vendor operate the workflow with the exported data, configuration, logs, and process documentation, or could it merely connect to the same systems? For many current AI SaaS deployments, full operational portability will be aspirational. That is still useful. It tells the buyer which dependency is being accepted.

Concentration risk versus fragmentation risk

Many CIOs are not trying to add more composability. They are trying to reduce tool sprawl. That matters because the optionality argument has a serious counterargument: one accountable vendor can be less risky than a scattered stack of tools with fragmented ownership.

Incumbents know this. SAP, Microsoft, Salesforce, ServiceNow, and others pitch unified AI platforms as risk reducers. They are not always wrong. A single vendor can reduce coordination cost, simplify procurement, standardize identity and permissions, unify data models and support, and give the buyer one accountable party when outcomes fall short. But consolidation changes the risk. It does not erase it.

The trade-off is concentration risk versus fragmentation risk. Fragmentation risk asks: can we make all these tools work together, and who owns the outcome when they do not? Concentration risk asks: what happens when one vendor controls the workflow layer, the data context, the agent behavior, the pricing meter, and the roadmap? A buyer can rationally choose concentration. But it should be a conscious choice, not the accidental result of accepting the easiest AI bundle from an incumbent.

Pricing becomes a control surface

Traditional SaaS pricing was often frustrating, but it was legible. Seat counts, modules, contract tiers, and renewal uplifts gave buyers something to model. Agentic SaaS puts pressure on that model. If agents execute work that used to require named users, vendors will price less around access and more around usage, actions, resolutions, outcomes, or premium AI capacity.

That can match incentives. A vendor paid for resolved cases has a reason to improve resolution, just as a vendor paid for recovered revenue has a reason to increase yield. But the same model can create financial risk. If cost scales with agent activity, a successful deployment can create a budget problem. If pricing depends on outcomes, the buyer needs a clean way to define attribution. When the agent acts across several systems, who gets credit for the result? Who carries the cost when it makes an error? Outcome pricing becomes risky when the vendor’s commercial meter scales with the buyer’s operational dependence.

Pricing terms should become part of the governance negotiation, not a separate conversation:

Outcome-based pricing can be useful when the vendor truly owns part of the result and the contract makes accountability clear. It becomes risky when the vendor gets variable upside without accepting operational responsibility.

Contracting should match delegation

Procurement teams already know how to negotiate uptime, data export, termination rights, and renewal terms. Those still matter. But they miss the action-level concerns that appear when agents execute work. The contract needs to answer what the vendor can do, under whose authority, in which systems, up to what threshold, with what record, and with what recovery path.

Before giving an agent execution rights, the buyer should ask a more specific set of questions:

• What actions can the agent take?
• In which systems can it act?
• Does it act under a user identity, service identity, or vendor identity?
• What dollar, customer, legal, or operational thresholds limit its authority?
• What actions are prohibited?
• Where are action logs stored?
• Can the buyer pause the agent immediately?
• Can the buyer reverse an action?
• What happens if the vendor fails mid-workflow?
• Can the team run manually for 30 days?

These are not policy questions in disguise. They are operating requirements. An agent that can issue a refund needs refund thresholds, audit logs, exception rules, reversal paths, and dispute rights. One that approves expenses needs policy logic, fraud flags, approval records, human escalation, and liability allocation. One working a healthcare claim queue needs traceability, payer-rule updates, compliance controls, human review points, and recovery plans when automation fails. The deeper the delegation, the more the contract must move from software access to operational control.

Minimum governance before the vendor acts

Governance is often treated as a brake on AI adoption. In agentic SaaS, governance is what lets the buyer safely move from assistance to operational value.

Not every buyer can run an enterprise-grade AI governance program on day one. Many midmarket teams lack the architecture staff, procurement maturity, and data governance required for a full optionality review. That does not mean they should ignore the problem. It means they should tier it.

At a minimum, viable governance should cover the first set of risks before anything goes live. Most teams can start with these seven checks:

The buyer does not need to solve every future problem before signing. But the buyer does need to know which problems are being deferred. Use the grid before the contract is signed, not after implementation.

Start by placing the vendor in one cell based on what it will do on day one. Then place it in the cell the roadmap points toward. The gap between those two cells is where dependency grows unnoticed. A vendor may begin by orchestrating (improving prioritization and routing cases across teams) but the roadmap may point toward executing (updating records, issuing refunds, resolving cases, and reporting outcomes). That gap should trigger a different review. The buyer should negotiate for tomorrow’s operating role, not only today’s feature set.

The four questions for the next demo

The answer is not to avoid AI SaaS that absorbs work. Many buyers should use it. The value can be real: faster resolution, fewer manual updates, cleaner handoffs, better exception handling, and one accountable vendor.

The answer is to name the delegation before it becomes invisible. For the next AI SaaS demo, the test is not only “can the agent do the work?” It is these four questions:

• Show us the kill switch.
• Show us the action log.
• Show us the manual fallback plan.
• Show us the export path for workflow logic, not just data.

Until the vendor can answer those questions, do not let the agent execute. The test for AI SaaS buyers is simple: if the vendor stopped tomorrow, could the organization still perform the workflow, explain past decisions, control costs, and migrate the operating logic? If the answer is no, the buyer may still choose the vendor. But it should price, govern, and contract for that dependency before the vendor becomes the workflow.

As frontier models converge for a growing class of work, raw AI capability explains less of the difference between products. That is a narrower claim than it sounds. Model quality still matters. Accuracy matters. Latency matters. Long-context reasoning, code generation, multimodal performance, tool use, and safety behavior can decide whether a product works at all. But for many teams, the strategic question is no longer, “Can we get access to capable AI?” It is, “Can we turn that capability into a workflow users care about, trust, repeat, and eventually cannot replace?”

That is where product sense stops being a soft skill. In the AI era, product sense becomes product judgment: the operational judgment that decides where intelligence belongs, what context it needs, which data should enter the system, when humans stay accountable, and what durable asset the product compounds over time. AI strategy is expanding beyond model access into last-mile judgment, not moving away from models.

The feature that does not change the work

Picture a product team with access to the same frontier models as its competitors. The team ships an AI assistant into an existing B2B product. It can summarize records, answer questions, draft emails, and generate reports. The demo looks good. Users try it once. Then they go back to the old workflow.

The model did not fail. The product never changed the work. It sat beside the workflow instead of inside it. It did not know the user’s permissions, the current customer state, the exception history, the source-of-truth data, or the moment when a decision had to be made. It created another surface to check. This is a common failure mode when AI features are deployed without redesigning the work: impressive capability, weak product judgment.

A better team asks a different question. Not “Where can we add AI?” but “What recurring workflow is expensive, frequent, painful, or strategically important enough that intelligence should live there?” This shift matters because AI now touches nearly every part of the product lifecycle: discovery, design, prototyping, coding, testing, analytics, feedback, competitive analysis, sales, marketing, and operations. Open-weight models are improving. Enterprises increasingly use multiple models by use case. In a 2025 a16z CIO survey, 37% of respondents reported using five or more models in production. That statistic does not prove models are interchangeable. It does show that enterprise AI stacks are becoming multi-model, routed, and use-case specific. In that world, the ability to orchestrate models around proprietary context, workflow constraints, and measurable outcomes becomes a distinct source of value. The model becomes a component. The moat moves to the system around it.

Product sense is not taste

People often reduce product sense to taste: good UX instinct, customer empathy, prioritization, and a feel for what should be built. Those still matter. But AI stretches product sense into something more operational.

In an AI product, product judgment means deciding which workflows deserve intelligence, which context improves or degrades the answer, when the AI should act or stay silent, where human verification must remain, and which part of the value chain the company must own. This is why context has become a product decision, not just an engineering concern. Anthropic’s work on context engineering frames the problem clearly: agents perform better when systems curate what enters memory, what gets retrieved, what gets summarized, and what stays out. That is product judgment.

More context is not automatically better. A financial AI assistant that remembers every conversation may feel powerful until it uses stale preferences, exposes sensitive information, or treats a passing comment as a permanent instruction. The product has to decide what should be remembered, forgotten, hidden, surfaced, and audited. The same logic shows up in Notion’s AI strategy. A generic chatbot can answer questions. An agent with workspace context can help because it understands the documents, projects, owners, permissions, and history inside the user’s real environment. The value comes from connected context, not from the chat box.

Claude in Excel is another useful example. The opportunity is not “AI spreadsheet chat.” The opportunity is intelligence inside a painful, high-frequency workflow. A user is already modeling, reconciling, checking formulas, cleaning data, and preparing analysis in Excel. Asking that user to move into a separate AI interface adds friction. Meeting the user inside the spreadsheet changes the work. This is the product-judgment move: find the work as it really happens, then decide where intelligence reduces effort or improves judgment.

Data alone is not the moat

The easy counterargument is that product judgment matters less than proprietary data. Sometimes that is true. Incumbents with decades of enterprise data, distribution, compliance trust, and system-of-record status can have advantages a sharper startup cannot overcome by taste alone. In commerce, Amazon and Google have behavioral and advertising data that can shape agentic shopping experiences. In vertical AI, domain-specific interaction data and evals can make a smaller specialized system outperform a general one.

But “we have data” is not a strategy. Data creates advantage only when it is relevant, fresh, governed, accessible at the right moment, and tied to a feedback loop that improves the product. Raw data ownership does not help if the product cannot decide which data matters, how to retrieve it, how to protect it, and how to convert it into a better outcome. That is why the strongest AI products combine product judgment with structural assets.

A shopping agent does not win because it can generate a pleasant recommendation. It wins if it understands intent better than a browsing interface: price range, fit, size, availability, support, delivery timing, brand preference, return risk, and the shopper’s past behavior. Retailers can use clickstream data, carts, purchases, ignored results, and returns to understand which products matter in which contexts. The insight is not “add AI to shopping.” It is that shopping is full of hidden constraints. The product has to read them.

The same principle applies in enterprise software. Products that embed into core workflows, connect to proprietary data, and build workflow-native integrations create advantages that are harder to copy than a polished assistant. Products that generate valuable usage data can improve their evals, tune workflows, reduce errors, and create better automation loops. Taste without asset creation is fragile. A good AI wrapper can become a durable company, but only if it compounds into something model providers or incumbents cannot easily absorb: trusted execution, workflow position, proprietary data, distribution, domain-specific evals, implementation knowledge, or an agent-readable system.

The new strategic question: what must we own?

A company building with AI now faces a more precise build-versus-buy decision. It can buy models, infrastructure, and open-weight systems. It can route tasks across vendors, expose tools through MCP servers, build custom agents, wire proprietary data into an existing model, or hire forward-deployed teams to turn customer needs into working systems. The strategic question is not “Which AI should we use?” It is “Which part of the value chain must we own for this product to become defensible?”

For some products, the answer is the data layer. An agent-readable data layer lets AI clients read, query, reason about, write to, and connect data without scraping a human UI. That can become a structural advantage as agents become a normal way users interact with software. For others, the answer is workflow integration. If models are good enough for the task, the company that owns the workflow, permissions, history, and user trust may win. For regulated industries, the answer often includes governance. Healthcare AI adoption can fail when a system disrupts clinical workflow, adds documentation burden, or creates unclear accountability. In finance, legal, healthcare, and insurance, trust, auditability, retention, privacy, and policy enforcement are product features. They decide whether the product can be deployed.

For enterprise AI, the answer may be implementation. The spread of forward-deployed models shows a practical truth: many customers do not have an AI technology problem. They have a business implementation problem. They need the system to fit their workflows, data locality, latency requirements, compliance rules, approval paths, and outcome targets.

This also cuts the other way. Incumbents already own many workflows. Model providers are moving into memory, retrieval, evals, tool-use frameworks, and enterprise services. If they own more of the surrounding system, the last mile does not automatically belong to startups or application companies. Last-mile judgment is a test of where advantage can actually compound, not a slogan for the application layer. The value comes from how models, tools, data, agents, human judgment, and business processes work together inside the customer’s actual environment.

Product judgment also creates speed

There is another reason product judgment matters more as AI gets faster: bad paths get cheaper too. AI lets teams generate prototypes, write code, test variants, analyze feedback, and produce collateral faster. That speed helps only if the team knows what to ignore.

Strong product judgment compresses decision time. It helps a team eliminate the chatbot bolted onto the side of the product, reject the custom model when existing models are sufficient, avoid collecting context that creates privacy risk, and choose a narrow workflow agent with clear escalation rules instead of a general agent with vague authority. Product judgment does not compete with speed. It makes speed useful. Without judgment, AI accelerates activity. With judgment, it accelerates learning.

The last-mile AI strategy test

Before investing in an AI product bet, ask five questions. These questions separate an AI feature from a potential product advantage.

1. Workflow

What recurring workflow are we changing? The workflow should be frequent, expensive, painful, risky, or strategically important enough to matter. If the AI does not change how work gets done, it will compete with every other assistant for attention.

Weak answer: “Users can ask questions about their data.” Strong answer: “Analysts spend three hours every Monday reconciling forecast changes across spreadsheets, CRM exports, and finance assumptions. The product identifies discrepancies, explains variance, and prepares the review packet inside the spreadsheet workflow.”

2. Context

What does the AI know here that a generic model would not know? This can include user history, company data, files, permissions, behavior, domain rules, transaction state, workflow timing, or live system data. But context needs boundaries. The product must decide what to include, persist, retrieve, summarize, hide, or forget.

Weak answer: “We connect the model to all company data.” Strong answer: “For this renewal workflow, the AI sees the contract terms, support history, usage trend, approval policy, and current account owner. It does not see unrelated employee records, stale notes, or private conversations that cannot improve the decision.” A generic assistant has capability. A product with the right context has relevance.

3. Judgment

Where do humans still need taste, accountability, verification, or override authority? This question matters most in high-stakes workflows. AI can expand analysis, draft options, find patterns, and recommend actions. But accountability stays with people and institutions.

Weak answer: “The system decides automatically unless the user stops it.” Strong answer: “The system drafts the recommendation, cites the inputs, flags uncertainty, and routes exceptions to the accountable owner before anything changes in the system of record.” A decision product needs guardrails, escalation paths, audit trails, and clear authority. It should know when to act, when to ask, when to explain, and when to stop. If no one can reconstruct why the system made a recommendation, the product is not ready for serious work.

4. Asset

What durable advantage compounds over time? The answer cannot be “better UX” alone. The product needs an asset that gets stronger with use. That asset could be proprietary data, usage loops, domain-specific evals, workflow integration, trust, distribution, implementation knowledge, agent-readable infrastructure, or customer-specific outcome history.

Weak answer: “Our interface is easier to use than the generic assistant.” Strong answer: “Every completed workflow improves our exception library, eval set, approval logic, and customer-specific outcome history, making the next recommendation more accurate and easier to audit.” This is where many AI wrappers fail. They solve a visible pain, but they do not accumulate anything structural. A competitor with better distribution, deeper data, or model-level access can copy the surface.

5. Outcome

What measurable customer result improves? The answer should be concrete: time saved, revenue gained, errors reduced, decisions improved, compliance strengthened, work removed, cycle time shortened, or margin improved.

Weak answer: “The product makes the team more productive.” Strong answer: “The product cuts the weekly reconciliation cycle from three hours to thirty minutes, reduces formula errors, and gives the finance lead an auditable packet before the forecast meeting.” This question protects teams from demo theater. A product can look intelligent and still fail to change the economics of the customer’s work.

A strong AI strategy can explain exactly which workflow changes, why the product has privileged context, where human judgment remains essential, what asset compounds, and how the customer outcome improves.

The gateway question

The companies that win with AI will not be the ones that simply advertise AI. Many of the strongest AI products will disappear into the work. They will automate a painful step, improve a decision, expose the right context, or make a workflow agent-readable.

The gateway question is simple: what would have to become true for this AI feature to be missed if it disappeared? If the answer is “users would lose a convenient shortcut,” the product may still be useful, but convenience is easy to copy. On the other hand, if users would lose a workflow they now trust, a decision process they can audit, a data loop that improves with every use, or an outcome they can measure, the product is closer to a moat.

Before asking which model to use, ask which workflow deserves intelligence. Before adding more context, ask which context improves the outcome and which context creates risk. Before automating a decision, ask where human judgment, accountability, and override authority must remain. Before claiming a data moat, ask whether the data is fresh, governed, relevant, and used in a feedback loop. AI makes building easier. That does not make strategy easier. It moves strategy closer to the user, closer to the workflow, and closer to the judgment calls that decide what should exist in the first place.

A demo only has to impress once. An AI product has to work every day. The first version looks magical in a conference room: it answers the clean prompt, completes the happy path, and suggests that broader autonomy is one launch away. Then real users arrive. They ask incomplete questions, use old terminology, need exceptions, and care less about whether the system sounds intelligent than whether it handles the job correctly.

Autonomous agents are not a bad idea. The real lesson is that autonomy is the outcome of a workflow that has proved it can be trusted, not a launch setting. Most useful AI products begin by defining what the system should do, what it should not do, what good enough means, what evidence proves readiness, and when a human or another system must take over. The market sells AI as self-directed intelligence. Users trust it when it behaves like a well-designed workflow.

Here is the short version: before an AI product earns more autonomy, it should be able to answer five questions. What recurring user job does this product handle? What does good enough mean for the output? What are the known failure modes? What test or real-user signal proves the workflow is ready? What must be escalated to a human or another system? If the team cannot answer those five, the product is still a demo.

The failure is usually not the model

A familiar pattern has emerged in enterprise AI. Teams run a promising pilot, then struggle to turn it into production value. The cause is rarely one thing: poor data, weak incentives, unclear ownership, cost, change management, and model limits all matter. But one failure mode shows up again and again: the team treats AI capability as the product strategy. A model can generate a response. A product has to know which response matters, how to verify it, where to send it, what to log, what to retry, what to escalate, and what to do when the world does not match the prompt.

Imagine a customer operations team that wants to build an AI agent. The tempting version is a chat box that promises to resolve anything. It looks powerful because it has no visible boundaries. The better version starts smaller: it classifies the request, checks whether required context is missing, drafts a response using the right policy, identifies exceptions, routes uncertain cases to a person, and logs why each step happened. It starts with read-only access, then earns permission to take action after the workflow proves itself. That version looks less magical. It is also closer to a product.

Start with intent, not intelligence

The first question is not “how autonomous can this be?” The first question is “what work is this product responsible for?”

A useful AI product begins with intent infrastructure. That means the team defines the recurring user job, the required inputs, the desired output, the standard for good enough, and the line between what the system can decide and what it must escalate. This sounds basic because the perceived intelligence of the model makes teams skip basic work. When a system can answer almost anything, it is easy to mistake fluency for understanding the job.

A customer support agent cannot simply handle refunds. It needs to know which refunds are eligible, which markets have different rules, which requests require identity verification, which cases require empathy rather than speed, and which exceptions belong with a human agent. A financial analysis assistant cannot simply summarize risk. It needs definitions, thresholds, source rules, and a way to separate fact from inference. The more consequential the workflow, the more explicit the intent must become.

This is why strong AI products often start with narrow, repeated work. A personal assistant that analyzes a daily spreadsheet inside a clearly defined role is more product-shaped than a general assistant that promises to help with business. A Claude Code skill that walks a user through specific questions and reviews marketing advantages for specificity, maturity, and strength is more reliable than a vague prompt to “make this strategy better.” The narrowness is where the team learns what reliability costs, not weakness.

Reliability comes from structure

AI products are different from traditional software in one uncomfortable way: they can always produce something. That makes “done” harder to define. A conventional workflow often fails visibly: a button breaks, a field rejects input, a transaction does not complete. An AI system can fail more quietly. It can produce an answer that sounds plausible, misses a policy exception, cites the wrong context, or handles a rare case with misplaced confidence.

That is why workflow structure matters. Reliable AI systems use bounded steps, structured outputs, limited permissions, tests, evals, checkers, and fallback paths. They define what the system is allowed to do before it does it. They do not treat the prompt as the only control layer.

Anthropic’s guidance on building effective agents argues for starting with simple systems, evaluating them, and adding more agentic complexity only when simpler patterns fall short. Berkeley AI Research describes a related idea as compound AI systems, where the product’s performance depends on the surrounding system of models, tools, retrieval, orchestration, and controls. In other words, the harness is part of the product.

The model matters. But two products using the same model can perform very differently because one captures the right context, structures the work, tests the output, controls execution, and escalates uncertainty. The other sends a broad prompt into a broad interface and hopes the answer is good. Hope is not a product control.

The metrics can lie

Throughput is not trust. An AI support system can handle more tickets, reduce operational load, and still damage the experience in the cases that matter most. Volume tells you the system did something. It does not tell you whether the customer got the right outcome, whether the tone was appropriate, whether the edge case was caught, or whether repeat contacts rose after the interaction.

Before scaling, a workflow-first product asks: which cases are simple enough for automation? Which cases require human judgment? What signal tells us that quality is deteriorating? What repeat-contact rate, escalation rate, complaint rate, or reviewer override rate would trigger a rollback? What does good enough mean for customers, not just for operations?

Intercom describes Fin as an AI agent, but its public product framing emphasizes support channels, handoff paths, reporting, and review loops. That does not make it less agentic. It makes the autonomy legible. The useful distinction is between an unbounded agent promise and workflow-shaped autonomy, not between workflow and agent.

The product is the harness

A good AI product does not only answer. It manages the conditions under which an answer becomes useful. The harness decides what context reaches the model, what tools the model can use, the output structure, whether generated text becomes a draft, a recommendation, an action, or an escalation. It records what happened. It gives humans a way to correct the system.

This is why coding agents work best when wrapped in structured task lists, one feature per session, repository principles, tests, browser checks, and documentation. The agent appears autonomous, but the product experience depends on the scaffolding around it. The same pattern applies outside software.

A regulated AI product should define its domain of operation before implementation starts. It should decompose autonomous functions into bounded components and map each component to safety tests, inspections, documentation, and operating rules. In high-risk contexts, this is no longer only good design practice. Regulations such as the EU AI Act increase the pressure for logging, transparency, documentation, and traceability. That pressure points to the same product truth: if the system cannot explain its operating boundary, record its behavior, and show how users should interpret its output, it is not ready for serious autonomy.

The real counterargument is speed

The strongest objection to workflow-first AI is that workflows sound slow, not that they are wrong. That objection deserves respect. Some products can tolerate broader autonomy earlier: consumer tools, coding assistants, internal prototypes, and AI-native products often operate in environments where errors are reversible, users expect imperfection, and expert users can inspect the output.

A startup may also need to learn in public. A rival may ship a rough agent, collect feedback, and improve faster than a team that waits for a perfect harness. In low-stakes categories, that trade-off can be rational. The mistake is turning that exception into the default rule.

The better response to leadership is not “we cannot ship until every control is perfect.” It is “we can ship the amount of autonomy that matches the risk. If the error is reversible and visible, we can move faster. If the error is costly, regulated, or hard for users to detect, we need stronger evidence first.” That sentence changes the conversation. It does not frame workflow discipline as caution. It frames it as the mechanism that lets the team decide how fast it can responsibly move.

For enterprise, regulated, trust-critical, or high-consequence workflows, “move fast” does not remove the need for controls. It changes how quickly the team must build them. The practical rule is risk-calibrated: low-stakes products can expand autonomy faster. High-stakes products need stronger proof before autonomy expands.

The autonomy readiness test

Before asking whether an AI product can be more autonomous, ask whether it is workflow-ready. Use this test when evaluating a product idea, reviewing a roadmap, buying an AI tool, or deciding whether an agent should move from pilot to production.

The first five questions are the minimum: what recurring user job does this product handle, what does good enough mean for the output, what are the known failure modes, what test or real-user signal proves the workflow is ready, and what must be escalated to a human or another system.

Then add the operating questions that determine whether autonomy can safely expand. What inputs, context, systems, APIs, and business rules does the workflow need. What can the AI decide on its own. What permissions should the system start with. What behavior will be observed after launch. What signal would cause the team to restrict scope or pause automation. What evidence would justify expanding autonomy. What is the per-unit cost of the workflow, and when does it become economically viable. How will users know what the system can and cannot decide.

The list is product design for nondeterministic systems, not bureaucracy. A deterministic product needs tests because software breaks. An AI product needs tests, evals, monitoring, and escalation because it can appear to work while being wrong in ways users cannot easily detect.

What evidence-driven autonomy looks like

Autonomy should expand when the evidence supports it. That evidence can take several forms: eval pass rates above a defined threshold, real-user error rates below an acceptable ceiling, production interactions without critical failures, human reviewer override rates falling below a target, successful performance across messy edge cases, or a support team seeing fewer repeat contacts without worse customer sentiment.

The important part is that the evidence belongs to the workflow. General benchmark improvement does not prove that your customer support agent can handle billing exceptions. A model release note does not prove that your medical summarization tool can manage domain-specific terminology. A successful demo does not prove that your drive-through ordering system can handle noise, accents, jokes, interruptions, and malicious inputs.

That is the difference between a demo eval and a product eval. A demo eval asks “can the system do this once?” A product eval asks “can the system do this repeatedly, under real conditions, with visible failure handling, at an acceptable cost, without damaging trust?”

How to use the test in a product review

Picture a product manager reviewing a proposed AI feature. The team wants to launch an agent that handles customer operations. The demo is strong: it reads a ticket, checks policy, drafts a response, and closes the case.

The product manager should not start by asking whether the agent feels impressive. The review should start with the workflow. What categories of tickets can it handle? What policy sources does it use? What does it do when information is missing? What counts as a correct resolution? What cases must it route to a person? What metric would reveal that customers are unhappy even if ticket volume improves? What permission does it need on day one? What permission should wait?

Those questions turn the product from a claim into an operating system. The answer might still be ambitious. The team might decide that the AI can classify every ticket, draft responses for half of them, automatically resolve a narrow subset, and escalate anything involving account closure, legal terms, billing disputes, or emotional distress. That is autonomy with a map, not anti-autonomy.

The buyer’s version of the test

The same logic applies when buying AI products. “Agentic” has become a sales word. A buyer should ask whether the vendor’s autonomy is visible enough to evaluate. Is this actually an autonomous agent, or a bounded workflow with a conversational interface. What systems does it connect to. What permissions does it need. What does it log. What does it refuse to do. How does it escalate. What evals prove it works in our environment. What happens when our data, users, policies, or edge cases differ from the vendor’s demo.

These questions do not slow procurement. They reduce the chance of buying a proof of concept that never becomes a product. For enterprise buyers, workflow-first positioning is a strength. A narrow, observable, eval-backed deployment is easier for risk, legal, security, and operations teams to approve than a broad autonomy pitch with unclear controls. A simple rule helps: until a vendor can name what the system refuses to do, the product is probably not ready to earn more permission.

Gateways to autonomy

The path to better AI products does not run through less ambition. It runs through better gates. Before you ask whether the AI can act, ask whether the workflow defines the action. Before you scale the agent, ask whether you know what good enough means. Before you grant write access, ask whether read-only performance has earned it. Before you trust the output, ask whether the eval reflects real users, not clean demos. Before you celebrate automation, ask whether quality metrics agree with volume metrics. Before you expand autonomy, ask what evidence would also force you to contract it. Before you buy the agent, ask whether the harness is visible.

A magic trick rewards surprise. A product rewards repeatability. The best AI products do not become useful because they look autonomous on day one. They become useful because the team defines the work, builds the checks, observes the behavior, and expands autonomy only when the workflow proves it can carry the weight.

The easier it gets to add AI, the more valuable it becomes to know where AI does not belong.

Most teams no longer struggle to access models that can summarize, draft, classify, route, recommend, answer, and act well enough to produce impressive demos. They struggle to turn that capability into something users trust, repeat, and pay for. That abundance creates a new failure mode. Teams start asking “where can we add AI?” when the better question is “where does AI improve the work?”

The last mile is everything between model capability and realized user value: the workflow where the model appears, the context it receives, the control the user keeps, the failures the product catches, and the outcome the business measures. As model capability becomes broadly available, durable advantage moves to the product layer: the harness, workflow, data context, trust boundary, and operating loop that turn model output into user value. AI strategy is won by knowing where AI should create value, where it should stay out of the way, and how to prove that it works better than the baseline, not by having AI. A wrapper shows that a model can do something. A product proves that the thing matters inside a real workflow.

The P&L gap

One of the most telling AI statistics is not a model benchmark. MIT Project NANDA reported that 95% of enterprise generative AI deployments produced zero measurable P&L impact. Treat the figure as directional, not as a universal law. The sample, definitions, and maturity of the deployments matter. Many AI projects are early experiments. Some were never designed to move a financial metric. But the pattern names a problem many leaders recognize: weak data, poor workflow fit, no clear business outcome, and no production measurement plan.

A CIO quoted in the research put it plainly: “We’ve seen dozens of demos this year. Maybe one or two are genuinely useful. The rest are wrappers or science projects.” The point is not that enterprise AI does not work. The point is that demos do not compound into value unless the product changes a real workflow and measures a real outcome.

Model access is becoming a weaker moat

For many horizontal use cases, model access is no longer the scarce asset it was two years ago. Inference costs have fallen. Frontier models have converged enough that the same base capability can produce very different user experiences depending on the product around it. One product makes the model feel like a natural extension of the work. Another makes the same class of model feel slow, vague, and risky.

Model choice still matters in regulated domains where an accuracy gap can decide whether a product is safe to use. Domain-specific models, fine-tuning, routing, and cost-performance judgment still matter when the cost of a wrong answer is high. But for many teams building AI into horizontal products, the harder question has moved up the stack: what job should the model do, in what workflow, at what quality threshold, with what human control?

Workflow fit beats feature presence

Imagine a product review meeting where every proposed roadmap item has “AI” in the name. One idea is technically impressive. It uses an agent to scan customer accounts, infer next steps, and generate a full action plan. In the demo, it looks powerful. In the actual workflow, it requires reps to leave their CRM, open a separate dashboard, review uncertain recommendations, and copy the useful parts back into the system where their manager measures activity.

Another idea looks boring. It cleans messy account notes inside the CRM, drafts a next-best follow-up, flags uncertainty, and asks the rep to approve before anything reaches the customer. The second idea wins, not because it uses a better model or has a more ambitious vision. It wins because it changes the user’s day without forcing the user into a new one.

This is where many AI products break. They assume users want intelligence as a destination. Most users want less friction in work they already have to do. GitHub Copilot’s success appears to come in part from tight workflow embedding: it appears inside the IDE, helps at the point of composition, and does not ask the developer to move the work somewhere else.

AI creates value when it meets the user where the work already happens. That does not mean every AI product must live inside an existing surface. Some new AI-native workflows will replace old ones. But even then, the product has to earn the behavior change. A standalone AI workspace has to be better enough to justify switching costs, training, governance, and new habits. Workflow fit is strategy, not polish.

The automation boundary is a product decision

Klarna’s customer service AI is a useful public case because it shows both the promise and the risk. Klarna said its AI assistant handled millions of customer conversations and automated roughly two-thirds of customer service chats. For simple questions, such as order status and payment schedules, that kind of automation can work well.

Then the story became more complicated. Klarna faced public scrutiny over the limits of automation and later described a need to rebalance human support in areas where customer needs are more complex. The story became more complicated later. The lesson here is that automation boundaries matter, not that customer service AI failed. In disputes, fraud claims, hardship cases, and sensitive financial conversations, confident but wrong answers about fees, policy, or payment terms are not merely annoying. They create compliance and trust problems.

AI can answer simple transactional questions. AI can draft responses. AI can route cases. AI can summarize context for a human agent. But not every customer problem should move through the same automation path. Good AI strategy requires deciding whether AI should suggest, draft, decide, act, escalate, or stay silent. That decision depends on user trust, risk, reversibility, cost of error, and the emotional weight of the moment. A model that works for password resets may be the wrong product choice for a fraud dispute. A system that can draft a clinical note may not be allowed to make a clinical judgment. An agent that can update a record may need approval before it changes anything that affects money, rights, or safety.

The boundary is also a learning decision. When AI acts without review, the product may lose the correction signal that teaches the system what good looks like. In some cases, the most defensible product will automate less at first, because human review creates the feedback loop the product needs to improve. These are product decisions, not just policy decisions. In AI products, product sense includes confidence, security, cost, auditability, training, and production readiness.

Almost right is still expensive

A model can win a benchmark and still lose the user. Benchmarks test capability under controlled conditions. Products succeed or fail under messy conditions: bad inputs, unclear user intent, slow response times, edge cases, partial trust, broken data, and users who are trying to finish work while being measured by someone else.

A traditional feature usually behaves the same way each time. AI features vary by prompt, context, user, retrieved data, model update, and confidence threshold. They can fail silently. They can be almost right. They can create support burdens that do not appear in a demo. The Stack Overflow developer survey data points to this trust problem: AI tool adoption has risen, while many developers remain skeptical of output accuracy and report frustration with code that is close enough to look useful but wrong enough to require careful review.

That is the danger zone for AI products. Obviously wrong output is easier to reject. Almost-right output creates hidden review costs. Those costs compound inside organizations. Every uncertain output shifts ambiguity onto the next human in the chain. The user has to inspect it, correct it, explain it, defend it, or absorb the risk of acting on it. If the product saves five minutes of drafting but adds ten minutes of verification, the metric that moved was activity, not value. The best AI teams treat launch as the start of measurement, not the end of implementation.

Product judgment shapes the moat

Proprietary data, feedback loops, distribution, bundling, business model, and regulatory positioning often look like separate advantages. In many cases, they are. Distribution especially can function independently of product quality. Salesforce, Microsoft, ServiceNow, and other incumbents can push AI features into products customers already use. That is a real advantage.

But distribution answers a different question. Distribution determines whether AI reaches users. Product judgment determines whether it earns repeated use, useful data, and trust. Data moats are also real, but simply having data does not guarantee a useful AI product. Teams still have to decide what data matters, how to collect it, how to refresh it, where to surface it, and how to turn it into a better decision for the user. When implementation gets easier, weak product judgment becomes easier to see.

The last-mile AI strategy test

Before building an AI feature, teams need a gate that is harder to pass than “the model can do it.” Use this test before committing roadmap, budget, or organizational attention.

1. User problem

What specific user pain, decision, or workflow does this improve? If the answer is “users want AI,” the idea is not ready. The problem should exist without the AI feature. The user should already spend time, money, attention, or risk on it.

2. Baseline

What does the user do today, how will you measure AI against that baseline in production, and who owns that measurement after launch? Do not compare the AI to a demo. Compare it to the actual workaround, spreadsheet, colleague, search process, support queue, or manual review path the user relies on now.

For a customer support drafting feature, that could mean measuring current handle time, escalation rate, correction rate, customer satisfaction, and policy errors before AI enters the workflow. The baseline does not have to be perfect. It has to be honest enough to prevent the demo from becoming the control group.

3. Workflow fit

Where does this enter the user’s existing work without creating new friction? The best AI products often feel less like a new destination and more like a better moment inside the current workflow. If the feature requires users to leave the system where work is measured, expect adoption to weaken. If the product is creating a new AI-native workflow, raise the bar. The new behavior has to beat the old one by enough to justify migration, training, and governance.

4. Automation boundary

Should AI suggest, draft, decide, act, escalate, or stay silent? For agentic AI, add a harder question: what does the human review before the agent acts, what can the user override, and what audit trail exists after the action?

5. Quality threshold

What reliability level is required before users trust it? Name the dangerous failure mode. In some contexts, a visible error is tolerable. In others, a silent or confidently wrong answer is unacceptable. The threshold should match the cost of being wrong.

A low-risk summarization tool can tolerate a different error profile than a fraud, clinical, legal, or payment workflow. Start with the cost of the error, compare against the current human or process baseline, then decide what level of review the system needs before the output matters.

6. Trust mechanism

How will the product show uncertainty, allow correction, preserve control, and recover from failure? Trust is a product behavior, not a message in onboarding. Users trust systems that make limits visible and give them control when the system is unsure.

7. Defensibility

What gets stronger over time? The answer might be proprietary data, workflow context, feedback loops, distribution, habit, or ownership of a measurable business outcome. If nothing compounds, the feature is likely a thin layer over model access.

8. Measurement commitment

What P&L or user outcome metric will this be measured against, who owns it, and what happens if the metric does not move within a defined window? This is where many AI projects reveal their weakness. They measure prompts sent, seats activated, or demos completed. Useful AI should move a real outcome: lower handling time with stable satisfaction, faster cycle time with fewer errors, higher conversion with lower manual effort, better compliance with less review burden. If the metric does not move, the team needs permission to change the workflow, narrow the use case, lower the automation level, or kill the feature.

How to use the test

The test works best before building, not after launch. Take a proposed AI feature and force it through the questions in sequence. Do not let the team skip from “user problem” to “defensibility.” Most weak ideas try to do that: they start with a vague pain, assume AI will solve it, then claim a data flywheel will appear later.

The stronger sequence is narrower. Start with one painful workflow. Define the baseline. Decide where AI enters. Set the automation boundary. Name the required quality threshold. Design the trust mechanism. Then ask what compounds if the product works.

This sequence changes the roadmap conversation. A team ranking AI use cases by executive excitement will choose visible, demo-friendly ideas. A team ranking by customer impact, technical feasibility, risk posture, business value, and measurement clarity will choose smaller use cases that can actually reach production. That difference matters. Many useful AI products begin with unglamorous work: cleaning inputs, drafting recommendations, routing cases, summarizing context, flagging uncertainty, or reducing review time. The value comes from making the work better, not making the demo louder.

Before adding AI to a surface, ask whether the user problem would matter without AI. Before choosing a model, ask what baseline the product must beat. Before building a standalone experience, ask whether the AI belongs inside a workflow users already trust. Before automating a step, ask what the human must review, control, or override. Before launching, ask what failure mode would damage trust fastest.

If the model gets ten times better, does this product become obsolete, or does the thing you own become more valuable?

Many companies are not building AI automation capabilities. They are rebuilding the same hidden machine in different departments.

Finance funds an invoice automation project. Legal buys contract AI. Product teams add a research synthesis tool. Compliance experiments with policy review. Each team writes a separate business case, evaluates separate vendors, defines separate workflows, and argues that its use case is special. On the surface, they are right. An invoice is not a contract. A customer interview is not a compliance filing. A missed payment discount does not carry the same risk as a bad legal interpretation.

But underneath, many of these projects converge on the same operating model: capture unstructured inputs, extract the facts that matter, classify them against a business taxonomy, validate uncertain outputs, and route the result into the systems where work happens. This matters now because scattered AI tools do more than duplicate spend. They create governance debt. Every isolated workflow adds its own review logic, audit trail, exception handling, vendor dependency, and learning loop. The longer each team treats its project as unique, the harder it becomes to scale AI without multiplying risk.

The answer is to identify the hidden machine, reuse the parts that genuinely repeat, and protect the domain-specific edge where expertise, risk, and judgment still matter. You do not have to pretend every workflow is the same to recognize that most of them share a core.

Three teams, one hidden machine

Picture three teams working in different parts of the company.

The finance team wants to process supplier invoices faster. Invoices arrive by email, portal, scan, and EDI. The system reads vendor names, purchase orders, line items, tax, and totals. Then it checks the invoice against purchase orders and receipts, flags duplicates, sends exceptions to review, and routes approved invoices into ERP for posting and payment.

The legal team wants to review contracts faster. Contracts enter from a repository or sales workflow. The system extracts clauses, obligations, dates, entities, and risk markers. It classifies contract type and risk level, checks deviations against the legal playbook, escalates exceptions, and routes the output into approvals, obligation tracking, renewal alerts, or CLM updates.

The product team wants to synthesize customer interviews and feedback. The inputs are transcripts, support tickets, surveys, call notes, and product reviews. The system extracts themes, quotes, pain points, requests, and sentiment. It classifies them by taxonomy, persona, journey stage, or urgency. Researchers validate the interpretation and source traceability. The output flows into backlogs, insight repositories, roadmap inputs, and stakeholder reports.

These teams think they are solving different problems. They are also building the same machine under different names. This does not make the domains interchangeable. Legal review needs legal playbooks. Finance needs matching rules and approval thresholds. Research needs source traceability and interpretive review. But the underlying operating model repeats often enough that treating each project as a blank-page build slows learning and scatters control.

The market is showing the pattern, not proving the platform

The shift is visible in how vendors and analysts describe the category. Gartner’s 2025 work on intelligent document processing frames the market around a pipeline that ingests content, extracts information, classifies it, and connects it to downstream systems. ABBYY and Camunda describe a similar split: document AI handles classification, extraction, validation, and human review, while process orchestration routes work forward. Hyperscience invoice examples follow the same pattern: classify, extract, validate, and feed structured data into financial systems.

The same logic is spreading beyond classic document processing. Customer feedback platforms ingest comments from many sources, structure them into product or customer themes, classify urgency or risk, and route alerts to teams. IBM’s Tokio Marine case shows customer feedback from calls and other channels being captured and classified into industry codes, with human approval reducing review time. Compliance monitoring examples apply a similar pattern to policies, filings, expense reports, contracts, obligations, risk categories, expert review, and remediation workflows.

These examples do not prove that every company should build one cross-domain platform. They show something narrower: the workflow grammar is repeating. Capture, extract, classify, validate, route. The pattern appears in enough places that leaders should stop evaluating each automation request as if it came from a different species of work.

Many AI workflows do not fit this pattern. Conversational agents, creative work, negotiation, strategy, and highly judgment-intensive decisions do not reduce neatly to a linear pipeline. But a large class of unstructured knowledge-work automation does converge on a shared architecture. Companies that see that pattern can stop buying the same capability in fragments.

The mistake is treating the domain as the architecture

A domain changes the rules of the workflow. It does not always require a new architecture.

An invoice project needs invoice schemas, PO matching, duplicate detection, tax logic, approval thresholds, and ERP integration. A contract project needs clause definitions, fallback positions, obligation models, legal playbooks, jurisdiction rules, attorney review, and CLM integration. A research synthesis project needs taxonomy design, quote verification, respondent context, bias checks, and links back to source material. Those are real differences.

But many of those differences live in configuration, validation, governance, and adoption. The reusable core is usually more basic and more durable: ingestion connectors, extraction infrastructure, classification services, review queues, confidence thresholds, exception handling, audit trails, observability, integration scaffolding, and governance patterns. The edge changes by domain: schemas, prompts, labels, model choices, thresholds, review protocols, compliance requirements, and routing targets.

This is the core-edge rule: standardize the machine, specialize the judgment.

The danger is oversimplifying it. Some domains require more than light configuration. Compliance, legal, healthcare, and financial workflows often need different extraction logic, model evaluation, escalation rules, and audit standards. In judgment-heavy work, domain expertise shapes the extraction and classification stages themselves. A legal risk model does not behave like an invoice field extractor. So the better claim is this: reuse the core where the overlap is real, and be explicit about where domain expertise must change the edge. It is not “build once and reuse everywhere.”

Validation is not a bolt-on

AI automation often stalls when validation is treated as cleanup. Validation is the control layer that makes an AI pipeline production-grade. It decides which outputs move forward automatically, which require human review, which need more evidence, and which should stop the workflow entirely.

In invoice processing, validation can mean three-way matching, duplicate detection, tax checks, and review of low-confidence fields. In contract analysis, it can mean checking a clause against a playbook, sending deviations to counsel, and preserving a trace from extracted obligation back to the original contract. In customer research, it can mean verifying that a theme is grounded in enough evidence, preserving the quote and speaker context, and preventing one loud customer from becoming a false product priority. In compliance, it can mean audit trails, rule checks, severity scoring, and escalation to a compliance officer.

Human review is not simply a safety net. It can become the bottleneck that destroys the business case. If every item falls into the review queue, the company has automated the easy part and preserved the delay. If reviewers lack clear thresholds, they become the system’s hidden logic. If corrections do not feed back into the pipeline, the same errors repeat.

That is why the validation layer has to be designed, not improvised. The workflow needs confidence thresholds, exception categories, escalation rules, reviewer roles, evidence trails, and a way to learn from corrections. It also needs throughput monitoring so the human review step does not become a permanent workaround. This is where many AI projects move from impressive demo to disappointing production system. The model can extract. The workflow cannot absorb uncertainty.

Capability building beats point solutions when the opportunity repeats

A point solution can make sense. If a workflow is narrow, isolated, and unlikely to repeat elsewhere, a focused tool may be enough. Specialized tools may also outperform a generic platform when the domain demands deep accuracy, certifications, or focused product investment.

But organizations with multiple document-heavy or knowledge-heavy workflows face a different problem. They are deciding whether to create a reusable way to automate many processes, not just one. The payoff changes when the pattern repeats. A reusable capability gives the organization shared intake templates, reference architectures, validation UI patterns, integration adapters, evaluation approaches, dashboards, governance checklists, and ROI calculators. Some assets stay highly domain-bound, especially schemas, prompts, taxonomies, and training sets. Others travel well, especially review queues, observability, audit trails, exception workflows, integration patterns, and delivery methods.

The second project starts with more than a blank page. The third project inherits lessons from the first two, as long as the organization is honest about which lessons transfer and which ones do not. Internal centers of excellence and platform teams can turn this into a repeatable delivery method: discovery identifies candidate workflows, volumes, error costs, cycle-time pain, decision points, and risk levels. Blueprint defines input sources, extraction targets, classification taxonomies, validation logic, routing destinations, governance needs, and operating roles. Build configures reusable components, prompts, models, review queues, integrations, dashboards, and audit trails. Pilot runs the system beside the current process, measures accuracy and throughput, tunes thresholds, and captures exceptions. Scale expands document types, increases straight-through processing where evidence supports it, monitors drift, and improves the operating model.

That is a different business than shipping one AI tool at a time. It is also harder. A shared platform can become a lowest-common-denominator tool. A central team can become a bottleneck. A single platform can create internal lock-in as surely as an external vendor can. The point is to make reuse an explicit architectural and operating decision instead of an accident, not to centralize everything.

The hidden machine diagnostic

The practical question is not “Can AI automate this?” That question is too broad. It invites demos, vague optimism, and tool-first thinking. A better question is: “Can this workflow be configured on top of a reusable automation core, or does it require a bespoke approach?” Use the hidden machine diagnostic to find out.

1. Capture

Identify the unstructured inputs: invoices, contracts, transcripts, emails, support tickets, forms, filings, policies, messages, scans, call recordings, or records. If the input sources are chaotic, optimize the intake process first. Automating a messy intake flow turns disorder into faster disorder.

2. Extract

Define the decision-relevant material: fields, entities, clauses, themes, risks, signals, anomalies, obligations, or quotes. The more interpretive the extraction, the less portable the configuration. The more traceability the workflow needs, the more deliberately the source evidence has to be preserved.

3. Classify

Map the extracted material to a business taxonomy: document type, risk category, customer segment, workflow status, urgency, obligation, severity, or topic. Rule-based classification often scales well. Judgment-based classification needs stronger review, evaluation, and governance.

4. Validate

Decide what must be checked before action: confidence scores, business rules, human review, exception handling, source traceability, duplicate checks, playbook checks, audit logs, or compliance review. This is the stage that separates useful automation from risky shortcut.

5. Route and act

Send the validated output where work happens: ERP, CLM, CRM, backlog, case system, insight repository, approval workflow, reporting process, remediation queue, or alerting channel. Routing is not clerical. It is where the automation becomes operational.

6. Learn

Feed outcomes and corrections back into the system. Human corrections, exception patterns, false positives, missed fields, reviewer overrides, and downstream outcomes should improve the workflow. If review produces no learning signal, the organization is paying people to catch errors without reducing future error rates.

7. Govern

Treat governance as a cross-cutting layer, not a final approval step. Governance includes audit trails, access control, retention policies, model and prompt change review, bias checks, drift monitoring, evaluation, ownership, and accountability. Standardization improves reuse, but it also concentrates risk. The platform that makes the tenth workflow cheaper is the platform that can break the first five with a single model update.

The real test: what repeats, what changes, what must be proved

The model works best when it forces separation. Do not ask whether invoice processing, contract review, and customer synthesis are “the same.” They are not. Ask which parts repeat.

The ingestion layer often repeats. Extraction infrastructure often repeats. Classification patterns often repeat. Validation queues often repeat. Observability and audit patterns often repeat. Integration scaffolding often repeats. Then ask which parts must change. The schema changes. The taxonomy changes. The prompts and models may change. The thresholds change. The reviewer changes. The audit burden changes. The risk tolerance changes. The route changes. The adoption problem changes.

Finally, ask what must be proved before scale. Accuracy is not enough. The workflow must prove that reviewers can manage exceptions, downstream systems can absorb outputs, governance can explain decisions, and users trust the result enough to change how they work. Forrester’s critique of AI adoption is right to push beyond technology architecture. Many AI programs fail because organizations do not redesign the work around the system. A reusable architecture helps, but it does not replace process redesign, operating ownership, training, or change management. The hidden machine is useful for many workflows. It is not sufficient by itself.

How to use the model before funding the next AI project

The next time a team asks for a new AI automation project, do not start with vendor selection. Start with gateways.

Is the process ready to automate? If the current workflow has unclear ownership, unstable inputs, inconsistent rules, or no agreed definition of success, fix the process first. AI will not rescue a workflow the organization cannot describe.

Does the workflow fit the hidden machine? Look for unstructured inputs, extractable decision material, a usable taxonomy, a validation point, and a downstream action. If those pieces are missing, the model may not fit.

What can be reused? Identify reusable ingestion, extraction infrastructure, classification patterns, validation queues, observability, governance, and integration scaffolding. If nothing repeats, a point solution may be the better choice.

What must be domain-specific? Name the schemas, prompts, taxonomies, thresholds, review protocols, compliance rules, and routing targets that cannot be generalized. Do this before anyone promises speed from reuse.

Where can the system act without review? Set confidence thresholds and exception rules early. If every output needs human approval, the project may still help, but it is a decision-support system, not automation.

Can the organization explain the result? If the company cannot reconstruct the source, rule, model output, human decision, and downstream action, the workflow is not ready for high-stakes use.

Will corrections improve the system? If human review produces no learning signal, the organization is paying people to catch errors without reducing future error rates.

What happens to the tools already running? If the organization already has scattered automation investments, do not rip them out first. Gate new requests, map the existing estate against the hidden machine, and look for renewal windows, integration points, or high-risk workflows where consolidation makes sense.

Does standardization create new risk? A reusable platform can reduce fragmentation and can also increase lock-in. The right question is which risk the organization can manage better.

The companies that get this right will not treat every AI workflow as a one-off experiment. They will build a shared operating model and then configure it carefully where the domain demands it. The real shift is from overlooking the hidden machine to deliberately engineering it, then tuning each domain edge with the rigor it demands. Not from human work to full automation, not from expert judgment to generic AI, and not from every tool to one platform.

When you work on AI in a regulated environment, you will hear the same question again and again: can the model say the same thing twice? The question is not wrong. It just aims too low. The real standard for regulated AI is whether the workflow can prove what happened.

Consider a finance team. They do not need a chatbot that can answer a question about revenue. They need to know which records the system could see, which permissions applied, which prompt version ran, which model generated the answer, which retrieval results shaped it, who approved it, and what became the official record.

A health insurer faces the same challenge. They do not need an algorithm that can recommend a care decision. They need to prove that a qualified human reviewed the evidence before the denial, delay, or modification of care. California’s SB 1120, the Physicians Make Decisions Act, now puts that principle into law for health plans that use AI or algorithms in medical necessity decisions.

In my experience, the conversation about AI governance keeps circling the wrong center. We keep asking whether the model is deterministic when we should be asking whether the workflow is reconstructable. Can the organization show what happened, why it was allowed, who had authority, and what became official? That is the Reconstructability Standard. The goal is to build workflows that make AI variation visible, bounded, testable, reviewable, and reconstructable, not to make probabilistic models pretend to be calculators. Regulated organizations will not trust AI because it is intelligent. They will trust AI when the workflow can prove what happened.

Why lowering the temperature is not enough

A common answer to AI unpredictability is: lower the temperature. That answer works for demos. It is too small for regulated work.

Lower randomness can help. Fixed model versions, fixed seeds, controlled infrastructure, deterministic tools, and careful sampling settings can make a system easier to test and debug. In some environments, they can make model behavior highly repeatable. But repeatability is not accountability.

Even if the model gives the same answer twice, the workflow around it may not. The source document may have changed. The user may have different permissions. The retrieval layer may return different evidence. A prompt or tool schema may have been updated. A human reviewer may edit the draft before approval. Production adds integration, security review, compliance checks, drift, broken tool contracts, downstream edits, and audit requests. A demo never has to survive that burden.

This is why AI pilots often stall at the handoff to production. The model may be capable, but the workflow cannot answer basic questions:

• Which facts did the AI rely on?
• Which version of the prompt shaped the output?
• Which tool call changed the result?
• Which human approved the action?
• Which policy allowed it?
• Which final record became authoritative?

If the organization cannot answer those questions, the system is not production-ready for regulated use. This is one of the most common failure modes I see: a capable model embedded in a workflow that cannot answer basic audit questions.

The three kinds of determinism, and which one matters

There are three different ideas that often get collapsed into one word. The first is model-level determinism: the model gives the same output for the same input. For LLMs, that is useful where teams can achieve it, but it should not be the only operating assumption.

Computational determinism means the same calculation produces the same result. In finance, actuarial work, optimization, and official business records, this still matters. If two users ask for the same approved revenue number, the answer cannot depend on location, timing, or prompt phrasing. In those cases, deterministic code or clear-box systems should handle the exact computation.

Workflow reconstructability means the organization can rebuild the decision context after the fact. It can show what inputs were available, what the AI produced, what controls ran, what humans decided, and what became the official record.

Regulated AI needs the third idea as a baseline. In some workflows, it also needs the second. The mistake is treating the model as the place where all accountability must live.

I want to be explicit about something that matters for audits. A model-generated explanation is not an audit trail. Chain-of-thought text is generated by the same probabilistic system as the answer. It may be useful as a draft explanation, but you should not treat it as verified evidence of what actually happened. The audit record has to live outside the model, in tamper-resistant systems the organization can inspect and defend.

It lives in the input packet, the evidence set, the prompt registry, the model configuration, the tool logs, the policy checks, the review decision, the approval record, and the final artifact. That is the artifact layer. It is where AI accountability becomes possible.

When the prompt replay is not enough

Imagine a compliance reviewer investigating an AI-generated recommendation that changed between Monday and Friday. Replaying the prompt is not enough.

On Monday, the user may have had access to one set of documents. By Friday, a source record may have changed. The retrieval layer may have returned different evidence. A policy prompt may have been updated. A tool schema may have failed validation. A model version may have changed. A human reviewer may have edited the draft before approval.

If the team only stored the prompt and response, the investigation stalls. What the organization needs is the whole workflow path: the original input and user instruction, the records and documents available at that moment, the permissions and retrieval boundaries in force, the prompt and policy versions, the model and configuration, the retrieved evidence, the tool calls and validations, the draft output before review, the human edits and approval decision, and the final record that entered the business system.

That is the Reconstructability Standard in practice. Not that the model will repeat the same words tomorrow, but that the organization can reconstruct why this outcome happened today.

Regulation is moving toward the artifact layer

This is no longer only an engineering preference. The EU AI Act requires high-risk AI systems to provide transparency for deployers, support human oversight, and maintain logs. NIST’s AI 600-1 Generative AI Profile names risks such as hallucination, data provenance, synthetic content, and human oversight as governance concerns for generative AI. Banking model risk expectations under SR 11-7 already push institutions toward conceptual soundness review, independent validation, ongoing monitoring, change management, and documentation.

Insurance is making the separation between AI output and human decision especially explicit. California’s SB 1120 requires physician or qualified health care provider oversight for medical necessity determinations involving AI or algorithms. New York’s Department of Financial Services has told insurers using AI and external data that they need governance, documentation, testing, and controls that explain how models function, what inputs they use, and how outputs affect decisions.

The direction is clear: regulators care less about whether an AI system sounds confident and more about whether the organization can explain, review, and control the path from input to outcome.

The UnitedHealthcare nH Predict controversy shows the risk. A Senate Permanent Subcommittee on Investigations report found that UnitedHealthcare’s post-acute care prior authorization denial rate rose sharply from 2020 to 2022 as the company expanded automation initiatives. Litigation has alleged that algorithmic predictions helped drive denials that were later reversed on appeal. Whatever the final legal outcome, the controversy illustrates the danger of insufficient transparency, weak oversight, and unclear authority when AI touches clinical coverage. In consequential workflows, the algorithm should not be treated as the decision by itself.

Keep the LLM on one side of the line

A probabilistic model can generate useful artifacts. It can draft a memo, summarize a claim file, extract fields from intake documents, propose inventory adjustments, identify missing evidence, or produce a structured recommendation.

But the workflow must decide what is true, safe, allowed, and final. That separation matters most when the output triggers a consequential action: a payment, denial, system-of-record update, legal filing, clinical coverage decision, credit decision, or regulated communication.

Consider an insurance claims intake workflow. A safe design does not ask an AI agent to approve or deny complex claims on its own. It asks the system to ingest documents, extract facts, flag exceptions, cite evidence, record confidence, route uncertain cases, and present a reviewable packet to a human decision-maker. The final disposition belongs to the authorized reviewer, not the model.

The same pattern applies to finance. An AI assistant can help a finance team find records, summarize variance, or explain a metric. But if the question asks for an official number, the LLM should not invent the answer through token generation. A deterministic calculation layer should compute the number from governed data. The AI can explain the result, but it should not become the source of truth.

The pattern also applies to law. An AI tool can draft a brief section or summarize precedent. But the filing becomes an official legal record only after citation validation, source checking, attorney review, and approval. Court sanctions for hallucinated legal citations show the danger of letting generation pass into the record without a validation layer.

Human review alone is not enough if it means a rushed rubber stamp. Meaningful review needs evidence, authority, time, training, workload design, and a recorded decision. The reviewer must be able to approve, reject, edit, escalate, or override the AI output under a defined policy. The artifact layer does not guarantee good judgment. It makes judgment visible enough to test, challenge, and improve.

Measure variation before you deny it

The goal is not to deny AI variation. You need to measure it before the workflow reaches production.

A regulated team should test the same workflow across repeated runs, realistic inputs, changed retrieval states, edge cases, permission differences, tool failures, and model updates. If a target field changes across runs when it should not, the workflow is not ready for that use case.

This is not only a model benchmark problem. Benchmarks can show capability. They do not prove the system will perform delegated work consistently inside the enterprise. A tool-calling agent can fail because the schema changed. A retrieval workflow can fail because the source boundary is too loose. A clinical support workflow can fail because the human approval step happens after the harm, not before. A finance workflow can fail because the model generated a plausible number instead of calling the governed calculation layer.

Variance testing should answer a practical question: is the variation acceptable for this workflow’s consequence level? For low-risk drafting, more variation is tolerable. For official records, financial outputs, legal filings, clinical coverage, and system updates, variation must be tightly controlled or moved out of the LLM.

The five questions a regulated workflow must answer

A regulated AI workflow is accountable enough when it can answer five questions without heroic investigation.

1. What shaped the output?

The workflow should preserve the input packet: documents, records, data, user instructions, permissions, system state, retrieval boundaries, prompt versions, policy versions, model configuration, tools, schemas, and runtime controls. If two users receive different answers, the input packet should show whether they had different permissions, different source access, different timing, or different instructions.

2. What evidence did the AI use?

The workflow should preserve the evidence set: retrieved documents, citations, records, facts, confidence scores, source boundaries, and validation results. For regulated work, sources are not background material. They are audit objects.

3. What happened during execution?

The workflow should preserve the execution path: tool calls, validations, policy checks, fallback flows, memory writes, system actions, blocked operations, and delegation chains. Agentic workflows need more than prompt-response logs. They need records of actions, state changes, failed operations, and the authority each agent or tool used.

4. Who reviewed and approved it?

The workflow should preserve the review path: the generated artifact, later edits, reviewer identity, approval decision, rejection, escalation, override, and governing policy. The record should show who initiated the action, whether the actor was human or AI, and why the action proceeded.

5. What became official?

The workflow should preserve the final business record: the approved artifact, decision, communication, update, filing, or system-of-record change. The final record should link back to the draft, evidence, controls, and approval path. A workflow that cannot be reconstructed is difficult to defend in a dispute, audit, incident review, or regulatory examination.

You do not need to store everything forever. You need to define the minimum evidence required to defend the workflow at the level of risk it creates.

Not every workflow needs the same controls

Not every AI workflow needs the same controls. A drafting assistant for internal brainstorming does not need the same artifact stack as an AI-assisted claims decision. A customer support summarizer does not need the same assurance as a system that updates a bank’s official ledger. A legal research assistant does not need deterministic computation for every sentence, but it does need citation validation before anything enters a filing. In my experience, a simple three-tier framework is usually enough.

Tier 1: AI-assisted drafting

The AI generates candidate text, summaries, or options. A human owns the final artifact. The main controls are source capture, draft comparison, review, and final approval.

Tier 2: AI-generated output with human approval

The AI recommends an action that affects a customer, patient, employee, counterparty, or regulated record. The main controls are evidence packets, policy checks, reviewer authority, approval records, escalation paths, and audit trails.

Tier 3: AI-linked consequential action

The workflow can trigger financial, clinical, legal, operational, or system-of-record changes. The main controls are deterministic computation where exactness matters, runtime enforcement, strong permissioning, mandatory pre-approval, rollback, monitoring, and incident response.

The framework does not make every workflow heavy. It helps teams stop applying light controls to heavy-risk work. That is the mistake I see most often: a team treats a Tier 3 workflow like a Tier 1 prototype because nobody named the difference.

Start with the artifact layer, not the governance framework

A governance checklist fails if nobody owns the records. Platform teams often own model access, logging, gateways, and tool permissions. Application teams own workflow design, user experience, and business logic. Compliance teams own policy interpretation, documentation, testing evidence, and regulatory response. Business owners own the risk of the final process. A production AI workflow needs all four.

The first move is not to build a massive new governance platform. Start by picking one workflow and defining its artifact layer:

• The input packet
• The evidence set
• The prompt and policy registry
• The model, tool, and schema versions
• The execution and validation log
• The review and approval record
• The final business record
• The variance tests required before release

That package can start as a sidecar log, a decision artifact store, a prompt registry, or an event stream tied to the existing application. The architecture matters less than the discipline: the workflow must create records while work happens, not after someone asks for an audit.

The platform can record the model call, but it may not know whether the output became an official record. The application can show the user journey, but it may not know whether a retrieved source was permitted. Compliance can define review standards, but it cannot manually inspect every AI output. The business owner can approve the use case, but only if the system shows the evidence path.

This is why AI governance cannot live only in a policy document. It has to live in the workflow.

The standard is controlled accountability, not perfect sameness

Before trusting the output, can we show which inputs shaped it? Before accepting the answer, can we show which evidence supports it? Before letting the AI act, can we show which authority it used? Before treating it as official, can we show the final record and its source trail? Before scaling it, can we show how it behaves across repeated runs? Those are the gateways. If the answer is no, the problem is that the workflow still cannot prove what happened. Intelligence is not the missing piece.

When you are responsible for putting AI agents into production, you will face a governance question that sounds like security but turns out to be about operating standards. A policy tells a team what an AI agent is allowed to do. It does not tell the team whether the agent deserves to keep doing it. That distinction matters more as agents move from demos into production, where a customer-support agent can resolve cases, issue refunds, update records, and escalate when customers show frustration, a software-development agent can open pull requests, change dependencies, and trigger deployment steps, and a finance or insurance agent can gather documents, apply rules, and recommend decisions that affect real people.

In each of these cases, the governance question is not only “Is this agent allowed?” The better question is “What evidence would make this agent worth trusting in production?”

Many organizations already have AI policies, committees, intake forms, and approval gates. In my experience, the gap is not in having a policy. The gap is operational: teams approve or deploy agents without defining what the agent must improve, what it must never do, what evidence will prove it is working, and when humans must intervene.

This does not mean permission controls are secondary. Security reviews, access limits, legal review, vendor risk checks, and compliance policies all matter. Granular access control, context-aware permissions, least privilege, and just-in-time authority can prevent real harm. The problem is that permission controls define what an agent can attempt. They do not define whether the agent is improving the workflow, interpreting policy correctly, escalating the right exceptions, or still deserves the autonomy it has been given.

Agents need governance that starts with a provable obligation: value created, behavior bounded, risk constrained, actions visible, controls tested, and ownership assigned. Call that the production autonomy standard: the risk-scaled evidence an agent must meet to earn, keep, expand, or lose the right to act.

Why permissions are not enough

Most organizations first encounter agent governance as a permission problem: securing system access, gating customer data, limiting tool use, approving external communications, or deciding whether the agent can execute transactions. Those boundaries are necessary. They are not sufficient.

Consider a support agent with permission to issue modest discounts. It stays inside that permission. It does not break into a system, exceed its refund limit, or access data it should not see. But it starts offering discounts in cases where customer history, fraud risk, pricing rules, or policy ambiguity should have triggered review. A better permission model might bind the discount to customer status, fraud checks, policy flags, or approval thresholds. The broader problem remains: the organization needs an operating standard that tests whether the agent’s decisions remain appropriate, not only whether its permissions were respected.

The same pattern appears in public examples. Klarna said in 2024 that its AI assistant was doing the work of 700 customer-service agents. By 2025, the company was again hiring humans for customer support after its CEO acknowledged that an overemphasis on cost had lowered quality. Public reporting does not prove that a different governance standard would have prevented that outcome. It does show the danger of treating efficiency as a sufficient production standard.

Air Canada offers the legal version of the same lesson. Its chatbot gave a passenger incorrect information about bereavement fare refunds. In 2024, a Canadian tribunal held the airline responsible for the chatbot’s statement. The agent did not need exotic permissions to create liability. It needed a policy-alignment standard, a failure definition, and a handoff rule.

This is where agent programs drift. They measure task volume, response speed, cost reduction, or hours saved, but they do not measure whether the workflow actually improved. They celebrate activity because activity is easy to count. Production trust requires a different standard.

For a software-development agent, the right metric is whether the agent improves merge-request-to-deployment time, defect rate, deployment frequency, review burden, rollback rate, and security quality, not lines of code generated. For a customer-support agent, the right metric is whether the agent resolves the right issues, escalates the right cases, protects user trust, and stays inside policy, not messages sent. For a regulated workflow, the right metric is whether the agent can operate inside legal, audit, data, and human-review constraints, not automation rate.

Business outcome matters. But it must sit inside a constraint envelope. In healthcare, financial services, insurance, employment, and other regulated contexts, an agent cannot optimize for speed or revenue if that optimization compromises rights, statutory duties, auditability, or required human judgment. The first useful governance move is therefore simple: define what improvement means, and define what the agent must not do to get there.

What makes agent governance different

Traditional software can still fail, but its behavior is usually specified through deterministic logic. Given the same input and state, the system should produce the same result. Governance can often focus on access, change control, testing, and policy compliance.

Agents change the standard because non-determinism combines with tool access, memory, retrieval, delegation, and action across systems. Their behavior can vary. Their mistakes can compound. A single bad interpretation can lead to several downstream actions before a person notices. That is why governance has to start before architecture hardens.

A team needs to define acceptable behavior. It needs thresholds for autonomy, risk boundaries, escalation rules, and observability from day one. It needs an answer to the question “What will we do when the agent behaves plausibly but incorrectly?” Agentic systems raise risks that ordinary software governance does not fully cover. OWASP’s Top 10 for Agentic Applications names risks such as goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, and cascading failures. These risks do not make agents unusable. They make vague approval dangerous.

A high-authority agent in a complex environment needs stronger constraints than a low-authority agent summarizing internal documents. An agent that can send external communications needs different controls than one that drafts but cannot send. An agent that can modify production systems needs least privilege, reversible actions, circuit breakers, and explicit approval thresholds. Governance starts by making those distinctions visible.

The production autonomy standard

Before an agent moves from pilot to production, teams need a shared artifact that translates governance into operating decisions. The point is not to create another document for a review folder but to force the right conversations before the agent has power.

A production autonomy standard is the risk-scaled evidence an agent must meet to earn, keep, expand, or lose the right to act. A low-authority internal research assistant does not need the same depth of review as an agent that can move money, alter customer records, or modify production systems. But every agent needs explicit answers to the same basic questions: what it is for, what it can do, what it must not do, how success will be measured, how failure will be seen, and who can stop it.

The lightweight version might be a one-page standard for an internal research assistant: outcome definition, allowed data sources, uncertainty disclosure, one human escalation path, and a quarterly review. The full version belongs on customer-facing, financial, regulated, security-sensitive, or production-changing agents: documented failure thresholds, traceability, sampled audits, human override, incident response, and named shutdown authority.

Proportionality matters. Without it, governance becomes a launch-blocking ritual. With it, governance gives low-risk agents room to learn and high-risk agents the controls they need. A practical standard should cover eight areas.

Purpose and scope

Define the workflow, user, task, autonomy level, tools, systems, data, memory, and external channels. State explicitly what is out of scope. The minimum viable criterion is that the team can state in plain language what the agent does, where it acts, what it can access, and exactly what it is prohibited from doing.

Business outcome

Define the measurable workflow result the agent must improve. Include the baseline, target KPI, operating cost, quality measure, risk measure, and user or customer outcome. The minimum viable criterion is that the team can name the outcome that proves the agent is worth operating and the result that would make it not worth continuing.

Permission boundaries

Define what systems, tools, data, actions, and transactions the agent can access or execute. Apply least privilege. Separate read, recommend, draft, approve, and execute authority. Bind the agent’s authority to the user, task, session, and risk level wherever possible. The minimum viable criterion is that the team can distinguish what the agent can see, suggest, change, approve, and execute without human approval.

Task performance

Define what good performance looks like. Include accuracy, completeness, usefulness, repeatability, evidence handling, policy adherence, citation quality, and acceptable human correction. The minimum viable criterion is that the team can evaluate outputs against a documented standard, not a general impression that the agent seems useful.

Failure criteria

Define unacceptable failure modes, error thresholds, policy violations, user harms, operational risks, and evidence that the agent is no longer safe or useful enough to continue. The minimum viable criterion is that the team can answer, before launch, what evidence would force restriction, suspension, redesign, or retirement.

Escalation criteria

Define when the agent must hand off to a human. Include confidence, complexity, ambiguity, customer sentiment, exception type, regulated decision points, financial exposure, and risk thresholds. The minimum viable criterion is that the team can identify which situations require human review, who receives the escalation, and how quickly they must respond.

Human control model

Define ownership, override authority, pause and deactivation rights, user appeal paths, exception accountability, and manual review requirements. The minimum viable criterion is that the team can name the human accountable for the agent in production and the humans who can override, restrict, or shut it down.

Monitoring and lifecycle decisions

Define the traces, logs, samples, evals, audit trails, review datasets, governance metrics, and decision cadence required from day one. Specify what evidence justifies launch, expansion, restriction, retraining, redesign, suspension, or retirement. The minimum viable criterion is that the team can show how production evidence will change the agent’s autonomy over time.

Why visibility is not optional

A success criterion you cannot observe in time to inform a decision is an aspiration, not a control. That is why success criteria become architectural requirements.

If the team says the agent must cite sources, the system needs to preserve source references. Escalation based on low confidence requires confidence signals, routing logic, and records of escalations. Avoiding sensitive data exposure requires access boundaries, monitoring, and logs that show what data the agent touched. Policy compliance requires samples, evals, and audit trails that show whether the policy held in real use.

Teams need traces for agent actions and model calls. They need conversation-level trace IDs, replayable interactions, real-user query datasets, and automated checks. They need deterministic tests for rules the agent must always follow. They need review of real usage, not only lab prompts. Without visibility, human monitoring becomes a ritual.

Let me be direct about why this matters. A person asked to approve an opaque agent decision is not exercising control. They are signing off without evidence. In regulated workflows, that “human in the loop” model is especially weak. If the reviewer cannot see why the agent acted, what data it used, what policy thresholds applied, and what authority it exercised, the review is not meaningful enough for consequential decisions.

Decision-level provenance matters most when the agent’s action has consequences. Logs that show timestamps, tool calls, and actors are useful. But they are not enough if they do not show the rule version, evidence evaluated, approval authority, and threshold applied at the time of action. Without that record, teams reconstruct compliance from fragments after the fact. An agent cannot be wrong and invisible at the same time.

Human in the loop is not a plan

“Keep a human in the loop” sounds responsible. In practice, it is often too vague to be useful. You need to specify which human, at what point, with what information, under what threshold, with what authority, within what response time, and for which actions.

A support agent might be allowed to answer routine questions on its own, but it must escalate when sentiment worsens, confidence drops, the customer asks for a refund above a threshold, or the case involves legal risk. A financial services agent might require a high confidence threshold before recommending certain actions, with lower-confidence cases routed to a trained reviewer. A general operations agent might use a target escalation range so routine work proceeds autonomously while high-risk exceptions receive review. The exact thresholds vary by use case. The important point is that the thresholds exist.

Human control also needs suspension logic. Escalation sends a case to a person. Suspension pauses the agent’s ability to act until someone clears it. Those are different controls. An agent that gives one uncertain answer might need escalation. An agent that starts sending confidential data into an unsecured workflow needs suspension. An agent that takes several unexpected actions across tools needs a circuit breaker. An agent operating in a regulated domain might need mandatory manual review for entire classes of decisions, regardless of confidence.

The organization also needs an owner. Not a committee in the abstract. A named accountable team or executive who owns the agent’s production behavior, approves scope expansion, accepts residual risk, and has authority to restrict or retire the agent.

This is not about pretending agents must be flawless where humans are fallible. Humans also misread policy, miss context, and make inconsistent judgments. The difference is scale and speed. An autonomous agent can repeat the same bad interpretation across thousands of interactions before the organization understands the pattern. Governance often fails because no one is answerable when the criteria fail. If no one can say who can turn the agent off, no one has governed the agent.

From approval to autonomy decision

The standard works because it turns approval into an autonomy decision. Imagine a SaaS company testing a billing-support agent. The agent can answer billing questions and issue refunds below a defined threshold. During a limited launch, it improves first-contact resolution and keeps refund errors below the sampled audit threshold. On routine cases, it performs well.

But the review shows a pattern. Escalations spike when customer sentiment turns negative. Audit samples show inconsistent policy interpretation on annual-plan proration and grandfathered pricing. The agent is not dangerous enough to shut down, nor ready for expanded refund authority.

A permission-only review might say the agent stayed inside its access rights. A production autonomy standard enables a better decision: keep the agent live for routine billing inquiries, restrict refund autonomy for churn-risk or high-value accounts, require human review above the refund threshold, tighten escalation rules, and review again after a defined operating window. Binary approval is not the goal. The decision is launch, restrict, monitor, expand, redesign, or stop, based on evidence.

What the standard changes about how you govern

The standard changes the order of governance. Instead of asking a committee to approve a vague agent concept, the team must define the operating evidence first. Instead of debating policy in the abstract, security can inspect tool access and privilege, legal can identify the constraint envelope, product can define workflow outcomes, operations can define escalation and incident response, compliance can require audit trails and documentation, and engineering can build traces, evals, circuit breakers, and reversible actions into the system.

This does not make governance lighter. It makes governance usable. It also exposes weak use cases early. If a team cannot name the workflow outcome, the agent is not ready. If the agent needs broad access to several systems but no one can justify the permission set, it is not ready. If humans are expected to supervise but cannot see why the agent acted, it is not ready. If a vendor-managed agent cannot provide the traces, controls, or contractual assurances required for the workflow risk, it is not ready for that level of autonomy.

That sounds restrictive. In practice, it helps teams move faster on the right work. A team that defines success criteria before launch can build the right observability from the start, test the right risks, avoid automating a broken process, give the agent enough autonomy for routine work while protecting high-risk decisions, and scale only after the evidence supports scaling. The alternative is familiar: a promising pilot, a vague approval, a production incident, then a scramble to reconstruct what happened.

Governance as a lifecycle, not a gate

A success criteria standard should act as an operating license, not another shelf document. Before launch, it tells the team whether the agent is defined well enough to build. During testing, it tells the team what to measure. At production review, it tells approvers what evidence to inspect. After launch, it tells operators when to expand, restrict, pause, or retire the agent.

That lifecycle matters because agents do not stay still. Usage changes. Models change. Tools change. Policies change. Users discover edge cases. Attackers find new openings. Vendors update systems. Workflows drift. A governance model that only approves launch will age quickly. The better standard is continuous: the agent must keep proving it deserves production autonomy.

When you put permission on one axis and production governance on the other, four patterns emerge. Low permission and low governance: a sandbox assistant where neither matters much. Low permission but high governance: an overcontrolled low-risk tool. High permission but low governance: dangerous autonomy where the agent has power without controls. High permission and high governance: earned production autonomy, which is the only sustainable target for agents with real authority.

The riskiest agent is not always the one with the broadest permission. The harder risk to spot is the agent with meaningful permission and weak operating governance, because it can keep producing plausible, consequential errors without triggering an access-control alarm.

Agent governance is the operating standard an agent must meet to remain in production, and it starts with a question every team should answer before launch: who can turn it off, and under what evidence would they do it?