The Agent Looked Great in the Demo. Now Ask It These 8 Questions.

Every vendor’s agent looks impressive in the demo. Almost none of them will tell you how it behaves on a bad day.

Over 40% of agentic AI projects will be scrapped by the end of 2027. That’s Gartner’s read after polling more than 3,400 organizations, and the reasons they list aren’t exotic: escalating cost, unclear value, and weak risk controls (Gartner). Not one of those failures shows up in a sales call. The label “agentic” has spread so fast it now tells you almost nothing about whether the thing in front of you is safe to point at your systems and your data. A polished demo and a production-ready agent look identical from the buyer’s chair. The difference surfaces later, usually at the worst possible moment.

Our team builds and breaks agentic systems for a living, and we keep seeing the same gaps open up during diligence. So here’s the list we work from. These are the questions to put to a vendor, or to your own engineers, before an agent gets anywhere near production. For each one I’ve noted what a good answer sounds like, because the question matters less than what comes back.

1. Autonomy and control: what can it do without a human, and where are the stops?

You’re not looking for “it’s fully autonomous.” That’s a red flag, not a feature. You’re looking for a clear map of which actions the agent takes on its own, which need approval, and how you move that line later.

A good answer describes configurable autonomy. The team can dial the agent’s freedom up or down per action type, and there’s a defined way to interrupt or roll back what it does. OWASP now ranks excessive agency and agent behavior hijacking among the top risks for autonomous applications for a reason. An agent you can’t throttle is an incident you can’t contain.

2. Permissions and blast radius: what does it touch, and what’s the worst case?

Treat the agent as you’d treat any other identity in your environment, because that’s exactly what it is. The right answer is built on least privilege: the agent holds the narrowest set of permissions its job requires, not a broad service account that can reach everything.

This isn’t hygiene for its own sake. Compromised agents move an estimated 16 times more data than a compromised human user (Obsidian Security), which turns a single bad decision into a bulk data event. Ask the vendor to walk you through the blast radius of a hallucinating or hijacked agent. If they haven’t thought about it, that silence is your answer.

3. Observability and auditability: when it acts, can you reconstruct why?

Non-deterministic systems make this harder and more important at the same time. You want full logging of the agent’s tool calls, its inputs, and the reasoning path behind a decision, retained in a form you can audit after the fact.

“It just works” is not observability. If you can’t answer “why did it do that” three weeks after the fact, you can’t run it anywhere regulated or high-stakes. The audit trail is not a nice-to-have you add later. It’s the thing that lets you tell an incident from a mystery.

4. Reliability and evaluation: how does the vendor know it works, and how will you?

A demo proves the agent can succeed once. Production asks whether it succeeds consistently across the messy inputs of real use, and the gap between those two is enormous. By some measures 88% of agents that work in a controlled demo fail when they hit real workflows (Prefactor).

The math is unforgiving. Chain a few steps together and small error rates compound fast: an agent that’s 70% reliable per step is only about 34% reliable across three steps. METR’s work on how long a task an agent can complete puts the frontier at a 50% success rate on multi-hour tasks, not the 99% you’d want before you stop watching (METR).

So look for an evaluation harness: a defined test suite, regression testing when the model or prompts change, and documented behavior at the edges. Ask specifically how they handle non-determinism. An agent that passes on Monday and fails on Thursday hasn’t been tested. It’s been sampled.

5. Security: has anyone tried to break it on purpose?

Agentic systems carry a class of risk traditional software doesn’t. An agent that reads untrusted content and can act on it can be steered by that content. Simon Willison calls the danger zone the “lethal trifecta”: access to private data, exposure to untrusted content, and the ability to communicate externally. Put all three in one agent and you’ve built an exfiltration tool that’s waiting for the right email.

This is not theoretical, and it is exactly what a standard test plan misses. In 2025, researchers found a zero-click flaw in Microsoft 365 Copilot (“EchoLeak”): a crafted email quietly instructed the assistant to pull data from OneDrive, SharePoint, and Teams and leak it, with no user click required. Salesforce’s Agentforce got hit the same way through a poisoned web-to-lead form (“ForcedLeak”). The pattern repeats across the incident timeline.

The answer you want is that the system has been through adversarial testing by people whose job is to attack it, not just validated by the team that built it. Those are different disciplines.

6. Data handling and compliance: where does your data actually go?

Map the path. Which model providers see what, what’s retained, where it’s processed, and whether any of that crosses a boundary your compliance team cares about.

A good vendor answers this without hedging, because they’ve already had the conversation with their own counsel. A vendor who waves it away has told you they haven’t.

7. Cost and failure modes: what happens when it goes wrong, and what does it cost?

Autonomous systems fail in expensive ways. An agent stuck in a reasoning loop burns tokens, and budget, until someone notices the invoice. Ask about guardrails: hard spend limits, loop detection, and graceful degradation when a dependency is down.

The bar is simple. The agent should fail safely and cheaply, not silently and expensively. If nobody can tell you what a runaway agent costs per hour, nobody has capped it.

8. Exit and portability: if you want to leave, can you?

Tie this back to architecture. If the agent’s integrations are welded to one model provider’s proprietary format, switching later means rebuilding them from scratch. Portability is worth demanding on day one, because it’s far cheaper to require up front than to retrofit under pressure.

This is where the diligence conversation quietly bleeds into the next one, about whether the whole integration layer is built on an open standard or a vendor’s walled garden. Hold that thought.

The pattern underneath the questions

Read the list again and a theme emerges. None of these questions ask whether the agent is clever. They ask whether the system around it is controllable, observable, and safe to put your name behind.

Here’s the part most pitches, and most checklists, won’t say out loud: passing all eight doesn’t make an agent safe. The model at the center is the softest link in the chain. Benchmarks that throw poisoned tool outputs at frontier agents still get them to misbehave roughly half the time (MCPTox). You are not going to diligence your way to a guarantee, because the guarantee doesn’t exist yet.

That reframes what diligence is for. You’re not proving the agent won’t misbehave. You’re making sure that when it does, the blast radius is small, the logs tell you why, the spend is capped, and you can pull it back. Containment beats confidence. The most mature teams we work with have stopped asking “can we trust this agent” and started asking “how much damage can it do if we’re wrong.”

It’s the difference between an interview and a reference check. You wouldn’t hand a new hire the keys to production because they interviewed well. The demo is the interview. These questions are the reference check, and they’re the part most agentic pitches quietly skip. An impressive agent and a deployable one are not the same thing, and the cost of confusing them lands a quarter or two after the signature.

We run agentic readiness reviews that put a system through exactly these questions, including the adversarial testing most vendors leave out. If you’re about to deploy an agent, or about to buy one, bring it to our team first. We’ll pressure-test it before your customers do.