Stop Your AI Agent from Failing: Top 10 Things to Check Before Deploying

Imagine this:
A logistics company built an AI agent to handle vendor order queries via email. It read the message, looked up the order status, and replied. The demo went well enough that the team shipped it. Three weeks into production, someone noticed the agent had been approving refund claims it had no business approving. A vendor had apparently worked out that if a refund request was phrased a particular way, the agent skipped the rules entirely.

Why might this have happened? Probably because nobody had asked the one question that mattered during testing: what happens if someone tries to fool this thing? That single missing check cost more than the agent saved in its first month.

Most agent failures look like this. The model works, and so does the demo. What is missing is the system around the model, the part that has to survive real users, real data, and inputs nobody anticipated.

Most guides on how to deploy AI agents cover the mechanics of shipping it. This article covers what you check first, because the mechanics are rarely what fails. We have used a Five-P framework to work through production readiness: Purpose, Permissions, Proof, Protection, and Performance. Each category carries two checks, for ten operational checks to clear before the agent goes live. The list applies whether you deploy AI agents on managed platforms or run them on infrastructure that you control. Let's get started.

What Makes an AI Agent Production-ready?

An AI agent is production-ready when its task boundary is documented, it runs with least-privilege permissions, accuracy is measured against a fixed evaluation set, it has survived deliberate adversarial testing, every action it takes is logged for audit, and a named person owns it after launch.

A successful pilot tells you the model can do the task. Production readiness tells you the system around the model can be trusted to do the task repeatedly, under load, in front of people who did not build it and who may not use it as intended. Most teams that deploy AI agents clear the first bar months before they clear the second.

The two often get confused because a pilot feels more satisfying because it produces a demo. Readiness produces a document, which is harder to celebrate and considerably more useful three weeks later.

Also read: How To Build AI Agents: From LLMs to Production-ready?

That is why testing an AI agent differs from testing a specific feature of that agent. Most teams deploy AI agents the way they ship a software feature: feed it inputs, confirm the outputs look right. That is useful, and it is not sufficient.

An agent is not a question-answering surface. It plans, calls tools, chains steps, carries context between them, and can run without a human watching each decision. As the shift from assistants to agents has accelerated across Indian enterprises, the gap between the two testing approaches has become the main source of production incidents.

A chatbot that answers incorrectly produces a bad answer. An agent that acts wrong causes consequences: a refund issued, a record deleted, an email sent to a customer, or personal data exposed to someone who should not see it.

A readiness review closes the gap between a bad answer and a bad outcome. That is also why general software testing habits aren't enough on their own: agent failure modes are specific, and others have already cataloged them.

So before inventing a checklist, it helps to know what already exists.

What Should an AI Agent Readiness Checklist be Based on?

Build an AI agent readiness checklist on published risk frameworks rather than assembling one from scratch. Three are directly relevant for teams that deploy AI agents into production -

1. NIST AI Risk Management Framework for governance structure

2. OWASP Top 10 for Agentic Applications 2026 for agent-specific attack surface

3. India’s DPDP framework for anything touching personal data.

The NIST AI Risk Management Framework organizes risk work into four functions: Govern, Map, Measure, and Manage. Its Generative AI Profile (NIST AI 600-1), published in July 2024, adds twelve risk categories specific to generative systems. NIST gives you the operating model: who decides, what gets measured, and how findings get acted on. It does not provide anything agent-specific, because the base framework predates tool-using agents running in production.

The OWASP Top 10 for Agentic Applications 2026, released in December 2025, fills that gap. It catalogs ten risk categories, ASI01 through ASI10, that emerge only when an AI system has goals, credentials, tools, memory, and the autonomy to chain them together. Where NIST tells you how to run the process, OWASP tells you what to look for.

For Indian deployments, a third layer exists, and it is the only one that carries obligations rather than recommendations. The DPDP framework is covered separately below, because its requirements attach to individual checks rather than to the checklist as a whole.

Neither NIST nor OWASP is written as a pre-launch checklist, and both are voluntary. The five Ps below reorganize what they cover into something a team can work through in an afternoon, with each pillar mapping to the framework material it draws on.

The ten checks that follow are grouped by pillar, in the order a team should work through them.

Purpose: Does the agent know what it is for?

Checklist #1. Is the task boundary documented?

Before anything technical, write down what the agent does and, more importantly, what it does not do. For example, "handles customer support" is not a boundary; it is a department. A usable boundary reads somewhat like this: answers order-status questions and processes refunds below a defined threshold without escalation, and refers every other refund request to a human.

Ask the prior question too, because teams routinely skip it: does this workflow need an agent at all? If the task is a fixed sequence with no judgment, a scripted workflow will be cheaper, faster, and far easier to audit. An agent earns its complexity only when the path genuinely varies by input.

Teams that deploy AI agents without a written boundary end up with scope creep by default, because the model will keep doing whatever seems reasonable in the context of the conversation in front of it. This check fails if nobody can produce a document stating what is in scope and what is out.

Checklist #2. Does the agent have a defined escalation path?

Every agent eventually meets a case it cannot handle. The question is whether there is a tested route to a human when that happens.

Test it directly. Hand the agent a case it was never designed to solve and watch what it does. An agent that flags uncertainty passes. An agent that improvises a workaround fails.

For decisions that are ambiguous, high-value, or hard to undo, route them to human review rather than letting the agent resolve them alone. OWASP catalogs the opposite failure as ASI09, Human Agent Trust Exploitation, where users over-trust what the agent tells them. An escalation path only helps if the person on the other end is genuinely reviewing rather than approving by reflex.

Permissions: can it touch more than it should?

Checklist #3. Does the agent run with the user’s permissions and not its own?

The convenient approach, and the most common one among teams rushing to deploy AI agents, is a service account with broad access. It works on day one, and it's why a single compromised session can expose an entire dataset. The agent should act with the permissions of the person who invoked it, and no more. If the user cannot see a record, the agent working on that user’s behalf should not be able to see it either.

Test it by running the agent under a deliberately limited account and confirming it cannot do anything that account cannot do. If it can, the permission model is decorative.

This is ASI03 in the OWASP list, Identity and Privilege Abuse, and it shows up in agent reviews more often than any other finding, precisely because the shortcut is so easy to take and so invisible once taken.

Checklist #4. Is there a hard limit on irreversible actions?

Some actions can be undone, such as:

• Drafting an email

• Flagging a record for review

• Generating a summary

There are other actions that cannot be undone (even if undone, it is not cheap) such as:

• Sending the email

• Issuing a refund

• Deleting a file

• Writing to a customer-facing system

The check is mechanical - Has someone gone through every tool the agent can call and sorted the available actions into reversible and irreversible? For anything in the irreversible column, add a confirmation step with a human, or hard-cap what the agent can raise on its own.

If that list does not exist, build it before launch. It usually takes an afternoon and is the highest-value hour in this checklist.

Proof: can you show the agent is ready?

Checklist #5. Has it been tested against a golden set instead of a handful of examples?

A golden set is a fixed collection of realistic cases with known correct outcomes, re-run every time the agent changes. Fifty cases is a reasonable floor for a narrow agent. It should cover the ordinary path, the edge cases, and the cases the agent is supposed to refuse.

Establish the baseline before the agent exists. Measure how the current process performs on the same cases, whether that process is a human team or a rules engine. Without that number, you cannot claim the agent is an improvement, only that it works.

Then separate weak evidence from strong.

The distinction matters because weak signals are not neutral. They actively create confidence that the strong signals would have corrected.

Checklist #6. Is there documented evidence and a named owner?

Before launch, the team should be able to produce the file, not the anecdote. That means:

• Golden-set results, including how many cases were run and the pass rate

• Adversarial testing results, including what was attempted and what held

• Cost and latency measured under realistic load, not in isolation

• Confirmation that tool access and permissions match the documented boundary

• Known limitations, written down rather than remembered

• A sign-off from someone accountable for the decision to launch

Ownership is the part teams often forget. An agent without a named owner is an agent nobody is watching. Write down who reviews the logs, who is alerted when the pass rate drops, and who has the authority to switch it off. If the answer to any of those is "the team", the answer is nobody.

Protection: what happens when someone tries to break it?

Checklist #7. Has someone actually tried to make it misbehave?

Functional testing asks whether the agent works. Adversarial testing asks whether it can be made to do something it should not. These are different exercises, and the second one is usually the one that gets cut for time.

At minimum, attempt the following:

• Prompt injection through content the agent reads rather than content the user types, such as instructions hidden in an email, document, or web page it processes (ASI01, Agent Goal Hijack)

• Instructions that try to widen the agent’s tool access or chain tools in an unintended sequence (ASI02, Tool Misuse and Exploitation)

• Inputs designed to corrupt whatever the agent stores between sessions, including retrieval sources (ASI06, Memory and Context Poisoning)

• Requests phrased to slip past a threshold rule, which is exactly what happened to the logistics agent above

The person doing this should not be the person who built the agent. Builders test the paths they designed. Your agent must be prepared for someone else to try paths they didn't think of.

Checklist #8. Is every action logged in a way a human can audit?

Model output is not a log. A usable agent log records what the agent did rather than what it said: which tool it called, with which parameters, on whose behalf, at what time, and what came back.

The test is a question, not a document. When something goes wrong three weeks from now, can someone reconstruct the sequence without asking the agent to explain itself? An agent’s account of its own reasoning is generated text, not evidence.

Log retention needs a decision, not a default, particularly where agent logs capture personal data. That decision belongs to whoever owns the agent under check 6, made before launch rather than during an incident.

Performance: does it hold up at scale?

Checklist #9. Do you know its cost and latency under real load?

An agent that costs a predictable amount per run in testing rarely holds that number in production. Once it handles thousands of requests a day, chains multiple tool calls, and retries failed steps, the total moves in ways a pilot never shows.

Split the cost into two lines before launch, because they behave differently.

• The variable line is model inference.

Token spend scales with volume, with prompt length, and with every retry. This line surprises teams. Control it by measuring cost per completed task rather than cost per API call, since a single completed task may involve several calls.

• The fixed line is the runtime layer.

It holds the agent loop, the queue, the tool servers, and any state carried between steps. It is the smaller line, but it is the one you can actually make predictable, and it matters more than teams expect.

Agent workloads do not fit the serverless default, because a function with a hard execution timeout will kill a long-running agent mid-task. Agents that plan across steps, wait on external systems, or run for hours need persistent compute rather than short-lived invocations, and agents that execute code or call internal tools need an isolated host.

A dedicated environment such as a Python VPS from Bacloud covers both, at a fixed monthly cost for the runtime layer.

Before launch, someone should be able to answer:

• What is the average cost per completed task, including retries and failures?

• What is the 95th percentile latency, measured end-to-end rather than per call?

• How do both move at three times expected volume?

Run these under realistic load. Cost and latency problems found in a load test are budget items. The same problems found in production are incidents.

Checklist #10. Do you have a plan for what happens when the model changes?

Model providers update models, deprecate versions, and change defaults. An agent tuned against one version can behave differently against the next, and the change will not announce itself. This is the check teams most often skip when they deploy AI agents once and treat the deployment as finished.

Decide the process before launch:

1. Pin the model version explicitly rather than pointing at a floating alias.

2. Re-run the golden set after any model or prompt change.

3. Compare the new pass rate against the previous one.

4. Set a drop threshold that triggers an alert rather than a discussion.

5. Re-check security, tool use, cost, and latency at the same time.

6. Keep a rollback path to the last version that passed.

Without this, model drift surfaces as user complaints, which is the most expensive way to discover it.

What the DPDP Framework Adds for Indian Teams?

Organizations that deploy AI agents touching personal data in India carry obligations that a generic readiness checklist will not surface.

The Digital Personal Data Protection Act, 2023 became enforceable when the Ministry of Electronics and Information Technology notified the Digital Personal Data Protection Rules, 2025 in November 2025. Enforcement is staggered: some provisions took effect immediately, while most operational duties for data fiduciaries fall on later dates over the next eighteen months. That is a runway, not a reprieve.

Three points where the DPDP framework intersects directly with the checks above:

• Purpose limitation meets check 1: The notice given to a data principal must state the purpose of processing in clear, plain language. An agent with an undocumented boundary is difficult to reconcile with a documented purpose.

• Reasonable security safeguards meet check 3: Safeguards are an obligation, not a best practice, and a broadly scoped service account is hard to defend as such.

• Breach notification meets check 8: Fiduciaries must notify the Data Protection Board and affected individuals of a personal data breach. Reconstructing what an agent accessed, and on whose behalf, requires the log described in check 8 to already exist.

For BFSI and other regulated sectors, sectoral obligations sit on top of this rather than beside it, so bring compliance into the readiness review early rather than at sign-off. None of this is legal advice, and teams processing personal data at scale should have counsel review the deployment rather than a checklist.

Who Owns Each Check?

One reason readiness reviews stall is that the ten checks belong to different people, and in most Indian teams that deploy AI agents, several of those roles sit in the same chair.

In practice, most of this lands on whoever sits closest to the deployment itself. If your team keeps discovering that nobody owns checks 5 through 8, the report covers that gap.

Our upcoming Q3 2026 eBook on The Deployment Gap will examine how Indian employers are actually defining that role, based on a study of Naukri and LinkedIn postings coded across four layers of agentic intensity. Stay hooked.

For teams building the capability internally rather than hiring for it, our Agentic AI course covers agent design, tool use, and evaluation, while the Generative AI course covers the evaluation and safety groundwork that checks 5 through 8 depend on.

How Do the 10 Checks Work Together?

They are not ten independent items. They form a dependency chain, which is why running them out of order wastes effort.

• Purpose gates everything

Until you write down the boundary, nothing can scope permissions, build an evaluation set, or define misbehavior to test for. Checks 3 through 10 all inherit their scope from checks 1 and 2.

• Permissions bound the blast radius

They determine how bad the worst case can get, which in turn sets how much effort the remaining checks deserve.

• Proof measures the agent against Purpose

A golden set is only meaningful as a test of the documented boundary. Without check 1, an evaluation set measures whatever the builder happened to think of.

• Protection tests whether Permissions hold under pressure

Adversarial testing shows whether the least-privilege model is real or aspirational.

• Performance confirms the whole thing survives volume and time

This is where cost surprises and model drift live, and neither is visible on day one.

Run them in this specific order. A team that red-teams an agent with an undocumented boundary is testing against its own assumptions.

What to Prioritize When You Deploy AI Agents on a Deadline?

Few teams clear every check before they deploy AI agents for the first time. Start where the damage is largest.

Purpose and Permissions come first. Vague scope and excessive access cause the worst outcomes and are the cheapest to fix. A day of writing and an access review removes most of the catastrophic tail.

Protection comes next, before the agent handles anything sensitive or irreversible.

Proof and Performance expand as usage grows. They matter more over time than at launch because they catch slow failures rather than loud ones.

Use this as a checklist, not a ritual. When an agent fails in production, the model is usually fine. The missing piece is a control nobody put in place.

Key Takeaways

• Document the agent’s task boundary, including what it must not do, before writing code.

• Ask whether the workflow needs an agent at all, or whether a scripted workflow would do.

• Run the agent on the invoking user’s permissions, never a broad service account.

• Sort every available action into reversible and irreversible, and gate the second list.

• Test against a versioned golden set, measured against a pre-agent baseline.

• Have someone who did not build the agent try to break it.

• Log tool calls, parameters, and the acting identity, not just model output.

• Measure cost per completed task and p95 latency under realistic load.

• Pin the model version and re-run the golden set whenever it changes.

• Name an owner with the authority to switch the agent off.

• Treat a successful demo as a starting point, not as evidence of readiness.

Conclusion

Teams that deploy AI agents into production need more than a working demo and a few correct test results. An agent reads business data, calls tools, and takes actions on other people’s behalf, which means its mistakes arrive as consequences rather than as bad answers.

The ten checks in this guide give a team a concrete way to review readiness before launch, not after an incident.

Start with the risks that are cheapest to fix and most expensive to miss: scope and access. Expand the rest as the agent takes on more work. The teams that avoid the refund-approval story are not the ones with the best model. They are the ones who wrote down what the agent was not allowed to do, then checked it.

Frequently asked questions

How long should testing take before an agent goes to production?

There is no fixed period. A workable rule is to build and run a golden set of realistic cases, probe the agent’s boundaries, and measure cost and latency under simulated load. A narrow agent with reversible actions may take a few weeks. An agent that handles personal data or irreversible actions will take longer, and the extra time goes to checks 7 and 8 rather than accuracy.

Who should be responsible for security testing?

Not the person who built the agent. Builders test the paths they designed, which is exactly the blind spot adversarial testing is meant to cover. A second person, ideally from security, should attempt prompt injection, permission escalation, and threshold evasion. Record what was attempted, not just the outcome, so the next reviewer knows what has already been ruled out.

What are the most common reasons AI agents fail after launch?

Unclear task boundaries, permissions wider than the task requires, evaluation based on a handful of examples rather than a fixed set, and no monitoring after launch. The pattern is consistent: the agent performs well on the cases someone thought of, and behaves unpredictably on the cases nobody did. Teams that deploy AI agents successfully differ from teams that do not mainly in how much they wrote down beforehand.

Does every agent need a human escalation path?

Any agent handling sensitive, high-value, or irreversible tasks does. Even a narrow agent will encounter cases outside its design. The more useful question is not whether an escalation path exists but whether it has been tested, and whether the person receiving escalations is actually reviewing them.

Do these checks apply to agents built on no-code platforms?

Yes, and in some respects more urgently. A no-code builder hides the permission model, the logging behavior, and the model version behind a configuration screen, which makes checks 3, 8, and 10 harder to verify rather than unnecessary. Ask the vendor what identity the agent acts under, what it logs and for how long, and how it handles a model version change. If those answers are not available, treat the agent as unsuitable for irreversible actions.