Building AI Agents Safely: Guardrails, Permissions, and Human Approval
May 25, 2026
AI agents are moving fast from demos to production systems. They can draft emails, query databases, trigger workflows, create tickets, and even take steps in business-critical systems with little or no human intervention. That power is the point—but it is also the risk. Once an AI agent is allowed to call tools, browse data, or act on a user’s behalf, security moves from “how accurate is the model?” to “how do we prevent unintended actions, misuse, and loss of control?” NIST’s AI Risk Management Framework emphasizes that trustworthy AI must be secure and resilient, and recent NIST work on AI agents specifically highlights the need to constrain and monitor agent access in deployment environments. (nist.gov)
Enterprises are adopting agents because they promise efficiency, scale, and better user experiences. But with that adoption comes a new class of operational risk: prompt injection, tool misuse, identity abuse, and actions that go beyond what a user intended. OWASP’s LLM security guidance calls out prompt injection as one of the most important threats, and its newer MCP guidance highlights token mismanagement, secret exposure, and privilege creep as recurring failure modes in agentic systems. (owasp.org)
The good news is that safe agent design is not mysterious. It is a layered discipline. You combine model-level safeguards, app-level controls, workflow-level approvals, least-privilege permissions, monitoring, and governance so that an agent can be useful without being dangerously autonomous. OpenAI’s function calling guidance, for example, emphasizes structured tool use, and its newer Agents platform adds tracing to support observability. Anthropic’s security guidance similarly recommends isolation and careful handling of tool execution. (help.openai.com)
1. Why AI agent safety matters now: the rise of autonomous workflows and enterprise adoption
AI agents matter now because they have crossed an important threshold: they are no longer just text generators. They are increasingly decision-and-action systems. A user can ask an agent to “check the customer’s record, update the ticket, and send a follow-up,” and the system may authenticate to internal tools, retrieve context, and execute the work. That shift from passive generation to active orchestration is precisely why security teams are paying attention. NIST describes AI agent systems as capable of planning and taking autonomous actions that affect real-world systems or environments. (nist.gov)
This matters because every new capability expands the blast radius of failure. A harmless hallucination in a chatbot is annoying. A harmless hallucination in an agent connected to HR, finance, cloud infrastructure, or a production database can become a business incident. The more an agent can do, the more it resembles software with delegated authority. That means the classic questions of identity, permissions, logging, and approval become central again—but now with a probabilistic model in the loop. NIST’s AI RMF frames trustworthiness around characteristics including security and resilience, which is a useful lens here: agent safety is not a niche concern; it is part of trustworthy deployment. (nist.gov)
Enterprise adoption accelerates the need for guardrails because organizations are trying to integrate agents into existing workflows, not build entirely new ones. That means agents may operate across legacy systems, SaaS apps, internal APIs, and data sources with uneven security maturity. OWASP’s LLM and MCP guidance reflects this reality by focusing on vulnerabilities that arise when model output is connected to tools and execution environments. In other words, the danger is not just the model; it is the model plus permissions plus context plus action. (owasp.org)
The practical takeaway is simple: if an agent can read, write, send, approve, delete, or execute, safety must be designed in from the start. Retrofitting controls later is expensive and often incomplete. Organizations that treat agents like ordinary chat interfaces tend to underestimate both the security and compliance implications. Organizations that treat them like semi-autonomous workers tend to build better systems.
2. Core risk model: prompt injection, tool misuse, identity abuse, and unintended actions
A useful way to think about AI agent safety is to separate risks into four buckets: instruction manipulation, tool misuse, identity abuse, and unintended actions. This model helps teams move from vague concern to specific control design.
Prompt injection is one of the best-known threats. OWASP defines it as malicious or misleading input that manipulates the model’s behavior, potentially causing data leakage, privilege escalation, or unethical outputs. In agent systems, prompt injection can be direct or indirect: the agent may ingest hostile instructions from a user message, a web page, a document, a ticket, or any other retrieved content. The core problem is that the agent may treat untrusted text as if it were legitimate instructions. (owasp.org)
Tool misuse happens when the agent calls the wrong tool, calls the right tool with the wrong parameters, or uses a tool in a way that creates damage. OWASP’s LLM guidance highlights insecure output handling, where model output flows into downstream systems without enough validation. In agentic systems, that means a tool call can become a command injection, a data leak, or an irreversible operational change if it is not checked carefully. OWASP’s MCP guidance also points to command injection and scope creep as recurring issues in environments where tools execute code or actions on behalf of the model. (owasp.org)
Identity abuse occurs when an agent inherits or impersonates a human or service identity too broadly. If the agent is allowed to act “as the user” with the user’s full access, then any prompt injection or logic error becomes much more dangerous. Short-lived, scoped credentials and strong secret management reduce this risk. OWASP’s MCP guidance explicitly warns about token mismanagement and secret exposure. (owasp.org)
Unintended actions are the most operationally painful failures because they may be perfectly valid actions that were simply not intended. The agent may send an email too early, close a support case prematurely, delete a file, or update a production record before a human had a chance to review the context. This is why safe systems separate “recommendation” from “execution” and build stepwise approval into the workflow. NIST’s current agent-security work points to constraining and monitoring agent access specifically because autonomous actions can have real-world consequences. (nist.gov)
In practice, these risks interact. Prompt injection can lead to tool misuse. Tool misuse can amplify identity abuse. Identity abuse can turn a simple mistake into a major incident. The strongest control strategy assumes layered failure, not single-point perfection.
3. Guardrails by design: model-level, app-level, and workflow-level safety layers
Good agent safety is layered. If you rely on a single safeguard, you will eventually find its failure mode. A better approach is to separate protections into model-level, application-level, and workflow-level guardrails so that each layer compensates for the others.
Model-level guardrails focus on how the model interprets instructions and produces output. Structured outputs and constrained tool calling are especially useful here. OpenAI’s function calling documentation notes that structured outputs with strict: true can make tool-call arguments conform exactly to a provided JSON schema, which reduces ambiguity and downstream parsing risk. That does not solve all safety issues, but it meaningfully narrows the space of malformed or unexpected tool arguments. (help.openai.com)
App-level guardrails sit in the orchestration layer. This is where you validate inputs, sanitize untrusted content, classify requests by risk, gate tool access, and verify that output is allowed before the action happens. OWASP’s guidance on prompt injection and insecure output handling is especially relevant here because the application layer is where untrusted text meets executable behavior. A strong app layer distinguishes between context the model may read and instructions the model may follow. (owasp.org)
Workflow-level guardrails define the business process around the agent. This layer determines what the agent may do automatically, what requires review, and what is never allowed. For example, an agent might draft a refund but not issue it, or prepare a deployment change but not push it to production. This is where you encode policy into the human process, not just the code. The benefit is durability: even if a model is upgraded, the workflow can still enforce the same approval logic. NIST’s governance and trustworthiness work reinforces the idea that AI risk management is broader than model accuracy; it includes the operational context in which AI is used. (nist.gov)
The best systems combine all three layers. The model keeps outputs structured. The app validates, filters, and logs. The workflow decides whether an action is merely suggested or actually executed. That combination makes agents far easier to reason about in production.
4. Permissioning and least privilege: read-only defaults, scoped tools, and resource boundaries
If guardrails define what an agent should not do, permissions define what it can do. The safest default for an agent is read-only access. That means it can inspect data, summarize context, and propose actions, but it cannot mutate records, send messages, or execute changes unless explicitly authorized. Least privilege is not just a security principle here; it is an engineering control that makes agent behavior predictable and reversible. OWASP’s MCP guidance specifically warns against privilege escalation via scope creep and emphasizes short-lived, scoped credentials. (owasp.org)
A strong permission model begins by splitting tools into categories. Read tools can inspect but not modify. Write tools can make changes, but only to bounded resources. High-risk tools may require extra approval or be reserved for humans entirely. For example, a customer-support agent might be allowed to look up orders and draft responses, but not issue refunds above a threshold. A devops agent might inspect logs and recommend remediation, but only a human can approve a production rollback.
Scoped tools are equally important. An agent should not receive a broad “admin API” when it only needs to update one object type or one project. Instead, it should get a narrowly defined capability tied to a specific resource boundary. This reduces the impact of compromise and improves auditability. NIST’s agent-security RFI explicitly calls for methods that constrain and monitor the extent of agent access in deployment environments. (nist.gov)
Resource boundaries should also be enforced at the environment level. Tool execution should be isolated from sensitive systems, secrets should not be exposed to the model unnecessarily, and the agent’s runtime should be separated from high-trust infrastructure. Anthropic’s security documentation recommends using virtual machines for script execution and tool calls, especially when interacting with external web services. That kind of isolation can reduce blast radius if the agent is manipulated or behaves unexpectedly. (docs.anthropic.com)
A useful rule of thumb is this: if an agent does not need it to complete the task, it should not have it. That principle sounds simple, but it is the foundation of safer agent design.
5. Human approval patterns: when to require confirmation, escalation, or dual control
Human approval is not a sign that AI agents are too weak to trust. It is a sign that some actions are important enough to deserve accountable review. The challenge is to apply approval intelligently, not everywhere. If every minor step requires a click, users will bypass the system or stop using it. If nothing requires review, the organization loses control. The best pattern is tiered approval.
Confirmation is appropriate for low-to-moderate risk actions. The agent prepares the action, explains it, and asks the user to approve before execution. Examples include sending a drafted email, creating a calendar event, or updating a non-critical record. This gives speed without full automation. OpenAI’s function-calling and structured-output approach supports this style well because the action can be represented clearly before it is executed. (help.openai.com)
Escalation is appropriate when the agent encounters uncertainty, policy ambiguity, abnormal inputs, or elevated risk. If the model is unsure whether a request is legitimate, if the tool output conflicts with expectations, or if the action touches a sensitive system, the agent should pause and route the case to a person with more authority or context. This is especially useful for customer disputes, legal questions, security incidents, and exception handling.
Dual control is appropriate for high-impact actions. In this pattern, one person proposes and another approves, or the agent requires two distinct approvals before proceeding. This is common in finance, production changes, access grants, and destructive operations. Dual control reduces the likelihood that a single compromised identity, prompt injection event, or mistaken approval leads to irreversible damage.
The key is to map approval patterns to risk level. A low-risk action may be auto-approved after policy checks. A medium-risk action may need user confirmation. A high-risk action may require manager, security, or compliance review. A very high-risk action may be blocked entirely for agents. NIST’s agent-safety work and AI RMF both support this kind of context-sensitive risk management rather than one-size-fits-all automation. (nist.gov)
Human approval is also an opportunity to improve explanations. The better the agent describes what it plans to do, why it matters, and what data it used, the easier it is for a human to make a good decision.
6. Monitoring and observability: logging, audit trails, anomaly detection, and red-teaming
If an AI agent acts in a system and nobody can reconstruct what happened, then the system is not truly safe. Observability is what turns agent behavior from a black box into something you can govern, investigate, and improve. NIST emphasizes AI security and resilience, and OpenAI’s Agents platform adds tracing precisely because traceability matters for debugging and accountability. (nist.gov)
At minimum, you should log the prompt, tool invocations, tool results, approval events, policy decisions, and final actions. These logs should be tamper-resistant and linked to an identity, a timestamp, a session, and a request identifier. In regulated environments, audit trails are not optional. They are the evidence that controls existed and were followed.
Anomaly detection is equally important. The system should flag unusual behavior such as repeated failed tool calls, unexpected access to sensitive resources, unusually large data retrieval, off-hours changes, or actions that diverge from typical patterns. This is especially helpful for catching abuse that slips past static rules. In an agentic environment, “normal” should be measured continuously, not assumed.
Red-teaming should be part of the operating model, not a one-time exercise. OWASP’s LLM security project exists because real-world attack patterns evolve quickly. Teams should test prompt injection, secret extraction, unauthorized tool access, risky multi-step chains, and malicious retrieved content. Anthropic’s work on browser-use prompt injection shows why this matters: once agents interact with untrusted content, attacks can be subtle and highly effective. (owasp.org)
A mature monitoring program answers three questions: What did the agent do? Why did it do it? And could we stop or reverse it if necessary? If you cannot answer those questions quickly, observability is not yet good enough.
7. Policy and governance: aligning AI agent controls with security, legal, and compliance requirements
Agent safety is not only a technical issue. It is also a governance issue. Organizations need policy that defines who may deploy agents, what data they may access, what actions they may take, how they are reviewed, and how incidents are handled. NIST’s AI RMF is useful here because it frames AI risk management as an organizational discipline, not just an engineering task. (nist.gov)
Security teams should classify agent use cases by risk. For example, an internal drafting assistant may be low risk, while an agent that can approve expenses or modify production systems is high risk. That classification should drive control requirements, including authentication strength, approval patterns, logging depth, and retention. OWASP’s focus on prompt injection, insecure output handling, and token mismanagement gives governance teams a concrete vocabulary for policy. (owasp.org)
Legal and compliance teams should review whether the agent processes personal data, confidential business data, regulated data, or exported data. They should also determine what disclosures are needed when an AI system is acting on behalf of a user. Governance should specify whether certain actions require human sign-off, whether outputs must be retained, and how long logs and traces are stored.
A practical governance model includes:
an approved use-case inventory,
a risk-rating process,
control baselines by risk tier,
incident response procedures,
periodic control reviews,
and a change-management process for model updates, tool changes, and policy edits.
This is especially important because agent capability changes quickly. A model upgrade can change behavior even when the application code stays the same. That means governance must be continuous, not periodic. NIST’s recent work on AI agents underscores that the security landscape is still evolving, which makes adaptive policy essential. (nist.gov)
8. Implementation blueprint: secure agent architecture, approval gates, and rollback controls
A secure agent architecture usually has five layers: user interface, policy engine, orchestrator, tool layer, and audit/monitoring layer. The UI gathers the request and shows the user what the agent intends to do. The policy engine decides what is allowed. The orchestrator plans and calls tools. The tool layer performs bounded actions. The audit layer records everything and feeds alerts to operations and security.
The important design principle is that the model should not directly control sensitive actions. Instead, the model proposes, while the system verifies. OpenAI’s function-calling and structured-output pattern supports this separation well because the tool schema can constrain what the model is even allowed to ask for. (help.openai.com)
Approval gates should be placed before any irreversible action. A secure workflow often looks like this:
The agent gathers context.
The agent drafts a plan.
The policy layer classifies the risk.
If needed, the agent requests human approval.
The system executes only after approval.
The system logs the action and outcome.
The rollback mechanism is prepared in case remediation is needed.
Rollback controls are critical. If an agent creates a bad ticket, sends an incorrect message, or changes a record, you need a way to undo or compensate. For mutable systems, that may mean soft deletes, versioning, delayed execution windows, or transactional safeguards. For external side effects, rollback may mean cancellation, correction, or human remediation. The safest systems assume failure and design recovery paths up front.
It is also wise to isolate execution from secrets and high-trust environments. Anthropic recommends VM-based execution for scripts and tool calls in sensitive contexts, which aligns with the broader principle of containment. Combined with scoped credentials and short-lived access, that architecture limits the damage if the agent is compromised. (docs.anthropic.com)
A secure blueprint is not about making agents slow. It is about making them safe enough to be widely useful.
9. Testing and evaluation: scenario-based evals, adversarial testing, and continuous improvement
You cannot secure what you do not test. For AI agents, testing must go beyond unit tests and model accuracy metrics. It should include scenario-based evaluations, adversarial cases, and continuous monitoring in production. NIST’s AI work emphasizes test, evaluation, validation, and verification as part of trustworthy AI, and OWASP’s projects exist because application-level failures are often revealed only through realistic abuse cases. (nist.gov)
Scenario-based evals ask, “What should the agent do in this situation?” For example:
A customer message includes hidden instructions.
A retrieved document asks the model to reveal secrets.
A tool response contains unexpected or malformed data.
The agent is asked to perform a high-risk action without approval.
The agent is given contradictory instructions from a user and a policy document.
Adversarial testing pushes harder. Red-teamers try to manipulate the agent into revealing data, bypassing approvals, escalating permissions, or taking harmful actions. OWASP’s prompt-injection guidance and Anthropic’s browser-use research show that these threats are not theoretical; they are practical and recurring. (owasp.org)
Continuous improvement matters because agent behavior changes over time. New tools are added. Policies evolve. Model updates shift outputs. New attack techniques appear. A mature program uses telemetry from real usage to update test cases, tune thresholds, and improve approvals. It also captures incidents and near-misses as new evaluation scenarios.
One effective pattern is to maintain a living “agent safety corpus” of bad examples and edge cases. Every incident, failed prompt-injection attempt, and blocked action becomes part of the test suite. That makes the system better over time instead of merely more complex.
10. Future trends: human-led agents, agentic security standards, and operationalizing trust
The future of AI agents is likely to be less “fully autonomous” and more “human-led, machine-assisted.” In practice, that means agents will continue to plan, draft, retrieve, and recommend, while humans retain authority over sensitive actions. This is a healthier framing because it aligns capability with accountability. NIST’s recent agent-security work suggests that operational controls around access, monitoring, and constrained deployment will become increasingly important as adoption grows. (nist.gov)
We are also likely to see more formal security standards and shared terminology for agentic systems. OWASP’s rapid expansion from LLM security into MCP-specific risks shows the field moving toward specialized guidance for connected, tool-using agents. That evolution matters because generic appsec guidance does not fully capture the risks of model-mediated action. (owasp.org)
Another trend is the operationalization of trust. Organizations will increasingly treat safety controls as measurable service properties: approval latency, denied-action rate, incident recovery time, audit completeness, and policy coverage. In other words, trust will become something you can observe and manage, not just claim in a slide deck. NIST’s focus on trustworthy AI, plus the growing attention to agent security, points in exactly that direction. (nist.gov)
Finally, expect better isolation patterns, better tracing, and better policy engines. The winning agent platforms will not be the ones that act the most aggressively. They will be the ones that can act safely, explain themselves clearly, and fail gracefully when they must. That is what operational trust looks like.
Conclusion
AI agents can deliver real value, but only if organizations treat safety as a first-class design requirement. The most important ideas are consistent across the board: use layered guardrails, keep permissions narrow, require human approval for risky actions, log everything, test aggressively, and align the system with governance requirements from the start. NIST, OWASP, OpenAI, and Anthropic all point toward the same core lesson: once a model can act through tools, the challenge is not just intelligence, but control. (nist.gov)
The practical goal is not to eliminate autonomy. It is to earn it. When agents are designed with least privilege, explicit approvals, strong observability, and rollback paths, they become far more usable in real organizations. Trustworthy agents are not those that never fail; they are those that fail safely, transparently, and within bounds. That is how AI agents move from promising experiments to dependable enterprise systems.