Why AI Security Is the Most Overlooked Part of Any LLM Deployment

Most LLM initiatives are evaluated on output quality and time-to-market. Security is treated as a late-stage compliance checkbox — something to be addressed after launch, if it surfaces at all. That sequencing creates a class of risk that is both underestimated and surprisingly inexpensive to address early.

The shift that changes everything is not the model itself. It is the agent.

When a language model is given a goal, a set of tools, access to external data, and the autonomy to act across multiple steps — the system stops being a chatbot and starts being an actor. It can read emails, query databases, execute code, call external APIs, write files, and coordinate with other agents. The blast radius of a single manipulated instruction expands accordingly. This piece examines the threat landscape that emerges at that point, grounded in peer-reviewed and preprint research published between 2023 and early 2026, and offers a practical architectural frame for teams deploying LLM-powered products in production.

The Trust Boundary Problem

Every time you embed an LLM into a customer-facing or internal workflow, you are introducing a new trust boundary. The question is whether that boundary is explicit and enforced, or implicit and assumed.

Traditional web applications have a relatively rigid input-output surface. The attack surface is well-understood: SQL injection, CSRF, auth bypass. The application does not interpret ambiguous natural language as instruction. LLM-based systems do — and that is precisely what makes them useful and dangerous in equal measure.

Greshake et al. (2023), in a foundational paper published at the ACM Workshop on Artificial Intelligence and Security, demonstrated that LLM-integrated applications fundamentally blur the line between data and instructions.¹ Their work introduced the concept of indirect prompt injection — a class of attack where adversaries embed malicious instructions into content that the model is likely to retrieve at inference time, without ever having direct access to the system. The data the model reads becomes the command the model executes. This was not a theoretical concern. The researchers demonstrated it working against real systems, including Bing Chat and GitHub Copilot, at the time of writing.

That paper was published in early 2023. By 2024, the attack surface had grown substantially. By 2025, it was structurally different.

What Changes With Agentic LLMs

A standard LLM deployment generates text. An agentic LLM generates actions.

The ReAct (Reason + Act) framework, which interleaves chain-of-thought reasoning with real-world tool invocations, has become a dominant pattern for building autonomous agents.² A ReAct agent operating in an enterprise context can: receive a goal in natural language, retrieve context from a knowledge base, call external APIs, write to databases, send communications, and loop through multiple iterations — all without a human in the loop.

This capability profile corresponds directly to what the research literature identifies as the “lethal trifecta”: privileged access to sensitive resources, exposure to untrusted input, and the ability to communicate externally.³ When all three conditions are satisfied, a single successful prompt injection can cascade into data exfiltration, infrastructure manipulation, or persistent compromise.

A 2025 empirical study evaluating 17 state-of-the-art LLMs — including GPT-4o, Gemini 2.5, and Claude models — found that 94.1% exhibited vulnerabilities to at least one attack vector across direct injection, RAG backdoor attacks, and inter-agent communication. A particularly significant finding was the vulnerability hierarchy: while 41.2% of models succumbed to direct prompt injection, 52.9% were vulnerable to RAG backdoor attacks, and a critical 82.4% could be compromised through inter-agent communication.⁴ Models that successfully resisted direct malicious commands executed identical payloads when the same request originated from a peer agent — revealing a structural flaw in current multi-agent security models.

The implication is clear: LLM safety training has primarily addressed human-AI interactions. AI-to-AI trust boundaries have not yet been adequately secured.

Four Attack Surfaces Every Deployment Team Must Understand

1. Prompt Injection: Direct and Indirect

Direct prompt injection occurs when a user deliberately crafts input to override the model’s system instructions. Indirect prompt injection is more insidious: the attacker poisons content the model will retrieve — a web page, a document in a knowledge base, an email — and the model, when it processes that content, executes the embedded instruction.

Research published in 2025 characterised this as the dominant real-world exploit vector for the 2024–2025 period, noting that OWASP’s GenAI Top 10 lists prompt injection as LLM01:2025, explicitly recognising that RAG and fine-tuning do not eliminate the risk.⁵ The OWASP GenAI Security Project released its Top 10 for Agentic Applications in December 2025, extending this further — in agentic contexts, prompt injection is no longer a jailbreak; it is a mechanism for redirecting autonomous decisions and triggering real-world actions.⁶

In August 2024, real-world attacks against enterprise platforms demonstrated four-stage kill chains: poisoned messages in shared channels influenced AI behaviour, causing it to extract and leak information from private channels through tool calls disguised as legitimate operations.⁷ The Slack AI incident and Microsoft 365 Copilot ASCII smuggling attacks in the same period established that RAG-dependent persistence — malicious instructions embedded in workspace artefacts like messages and documents — was not a theoretical construct.

For agentic systems, prompt injection is the initial access vector. Everything else follows from it.

2. Unsafe Tool Execution and Privilege Escalation

Agentic systems acquire their power from tools. They acquire their risk from the same source.

The GrantBox benchmark study evaluated LLM agents across real-world MCP (Model Context Protocol) servers and found that ReAct agents yielded an average attack success rate of 90.55% when faced with privilege-abusing scenarios. Even Plan-and-Execute agents — a more structured architectural pattern with improved resilience — showed an average attack success rate of 79.05%.⁸ The study concluded that current LLMs possess only foundational security awareness: they can occasionally identify overtly illegal privilege requests, but remain highly vulnerable to sophisticated manipulation.

Privilege escalation in agentic systems takes a form that differs from traditional software. An agent uses a combination of individually benign tools to achieve an action it was not explicitly granted. For example: using a file-read tool to access a credentials file, then using those credentials with an API tool to access a restricted system.⁹ No single tool call triggers an alert. The chain is the attack.

The Model Context Protocol, introduced by Anthropic in November 2024, has become the dominant standard for connecting LLMs to external tools. It has been adopted by major platforms and, within months, had over 5,000 community-developed servers. A January 2026 protocol-level security analysis found that MCP’s capability declarations are self-asserted without verification — a malicious server can claim any capability, and the client has no mechanism to validate those claims. The specification does not mandate capability enforcement at the message level, formally violating the Principle of Least Privilege.¹⁰

The security implications of MCP adoption at enterprise scale have not yet been fully internalised by most deploying teams.

3. RAG Pipeline Exploitation

Retrieval-Augmented Generation is now the standard architecture for giving LLMs access to current and domain-specific information. Gartner estimated that over 30% of enterprise AI applications use RAG as a key component.¹¹ It is also one of the most exploited surfaces in production LLM deployments.

RAG systems allow untrusted content — user input, web pages, uploaded files, third-party tool outputs — to flow directly into the context window of a model that has implicit authority over internal systems.⁵ A poisoned document in the knowledge base can silently redirect agent behaviour on every query that retrieves it.

The taxonomy of RAG-specific attacks has matured significantly. A comprehensive 2026 survey identified multiple attack families: corpus poisoning attacks that inject adversarial documents into the knowledge base; retrieval manipulation attacks that use surrogate models to craft queries shifting the stance of retrieved evidence; and data extraction attacks where instruction-tuned RAG systems can be induced to regurgitate datastore content verbatim.¹² The paper notes that secure RAG can be compromised by attacking the access path to external knowledge, without requiring any manipulation of the model itself.

For multi-agent RAG systems, the problem compounds. A 2024 paper on prompt infection in multi-agent systems demonstrated that compromised agents can propagate malicious instructions to other agents in the pipeline, even when agents do not publicly share all communications.¹³ The Morris II worm demonstrated in March 2024 showed this propagation pattern working as a five-stage attack — initial access, privilege escalation, persistence, reconnaissance, and lateral movement — all triggered through AI-readable text, with no executable malware.⁷

4. Inter-Agent Trust and Protocol-Level Vulnerabilities

As organisations move from single-agent to multi-agent architectures, a new category of risk emerges: the trust assumptions between agents.

Research from 2025 reports inter-agent attack success rates of 84.6%.¹⁰ A critical finding across multiple studies is that models apply different — and weaker — security filtering to requests originating from peer agents compared to requests originating from humans. This is not a prompt-engineering quirk. It reflects a structural gap in how current LLMs model authority and trust hierarchies.

Google’s Agent2Agent (A2A) Protocol, announced in 2025, enables communication between agentic applications regardless of vendor or framework — and introduces a new attack vector where systems can be deceived into routing all requests to rogue AI agents by lying about capabilities through exaggerated Agent Cards.¹⁴

The research community has characterised this moment in terms that practitioners should take seriously. A 2025 paper in ScienceDirect described the threat landscape as requiring formal definitions of protocol-level threat models, noting open challenges in MCP protocol vulnerabilities, agentic web interfaces, and memory-centric risks that current defences do not adequately address.¹⁵

Why Security Is Consistently Skipped

Understanding the attack surface is necessary but not sufficient. Teams that understand the risks still frequently under-invest in controls. The reasons are consistent across organisations I have observed in consulting engagements:

Security is framed as a deployment blocker. When security review sits at the end of a development cycle, it is positioned as a gate that slows releases. The rational response from product teams is to minimise its scope. Security integrated into architecture decisions from the start has no gate-keeping function — it shapes the design.

LLM-specific threat modelling is still immature. Most organisations apply traditional application security frameworks to AI systems. Those frameworks were not designed for systems where natural language is both the interface and the attack vector. The OWASP GenAI Top 10 (2025) and the Agentic AI Top 10 (December 2025) are the most complete public frameworks available, but adoption outside of specialist security teams remains low.

The failure mode is not immediately visible. A successful indirect prompt injection does not crash the application. It may generate a response that looks superficially correct while quietly exfiltrating credentials or forwarding sensitive documents. The absence of a visible error is mistaken for security.

Agentic AI is new enough that institutional knowledge is thin. Development teams understand how to build with LLMs. They do not yet have the same depth of security intuition they have developed for web applications over two decades. That gap is the window of vulnerability.

An Architectural Baseline for Secure Agentic Deployment

The research literature converges on a consistent set of architectural controls. None of them are exotic. Several align directly with principles that security engineers will recognise from traditional systems — applied to a new context.

Treat the LLM as Untrusted

The foundational shift is architectural, not prompt-engineering. Research has consistently demonstrated that an LLM cannot be fully trusted to police its own behaviour when faced with sophisticated adversarial input.¹⁶ Security should not rely on the model’s good behaviour. It should rely on the surrounding architecture, which enforces hard constraints independent of model state.

This means: even if the model is successfully injected, it can only execute a pre-approved plan, use tools scoped to the current task, and run code within a locked-down sandbox.

Enforce Least Privilege on Every Tool

The OWASP Agentic Top 10 identifies excessive agency as the highest-priority risk for agentic systems — an extension of the Principle of Least Privilege to autonomous tool-calling contexts.⁹ Concretely: agents should hold the minimum permissions required to complete their defined task, scoped per-session where possible, with no inherited ambient credentials.

The GrantBox study found that current LLMs have only foundational security awareness around privilege usage.⁸ The architectural implication is that privilege enforcement cannot be delegated to the model. It must be enforced at the infrastructure layer.

The AgentBound framework, published in 2025, proposed that MCP servers declare their access requirements explicitly in the form of generic server permissions, obtaining an auditable, enforceable operating-system-level policy that cannot be circumvented through prompt manipulation.¹⁷ This approach — manifest-driven access control — represents a practical operationalisation of least privilege for agentic systems.

Separate Data and Instructions

One of the most effective structural controls is architectural separation of trusted instructions from untrusted retrieved content. The Prompt Flow Integrity (PFI) approach, evaluated in 2025, demonstrated that enforcing data-flow constraints — identifying untrusted data, enforcing least privilege, and validating unsafe data flows — reduced attacked task rate from 57.99% (ReAct baseline) to 0.00% in AgentDojo benchmarks.¹⁸

In practice, this means classifying all content that enters the model’s context window by trust level, and applying different processing paths accordingly. Content retrieved from external, potentially adversary-controlled sources should never be treated with the same trust as verified system instructions.

Define Data Classification for Prompts and Outputs

Not all data should be permitted inside a model’s context window. Before deploying a RAG pipeline or agentic workflow, teams should define which data classes are permitted in prompts, which fields from internal databases the agent can retrieve, and what categories of information the system may include in outputs.

This is not a technical control in isolation — it is a governance decision that shapes every technical control downstream.

Add Deterministic Logging and Decision Traces

Strong observability is identified by the OWASP Agentic Top 10 as a non-negotiable security control for agentic systems.⁹ This requires detailed logging of goal state, tool-use patterns, and decision pathways — not just input/output logging at the API boundary.

Agentic systems operating without decision traces cannot be audited, cannot be forensically investigated after an incident, and cannot be monitored for anomalous behaviour. The latency cost is real. The alternative — operating blind in a system that can execute consequential actions autonomously — is not an acceptable trade-off in production.

Build Human-in-the-Loop Gates for High-Consequence Actions

For actions with significant real-world consequences — writing to production databases, sending external communications, executing financial transactions, provisioning infrastructure — agent execution should pause and require explicit human approval.¹⁶

This is not a limitation on agent capability. It is an architectural acknowledgement that autonomous systems operating under uncertainty require human checkpoints on decisions with asymmetric downside risk.

Implement Defence-in-Depth

No single control is sufficient. Research on prompt injection defences has consistently found that state-of-the-art defences can be bypassed by attacks more sophisticated than those seen during training.¹⁵ ML-based detectors are susceptible to optimisation-based adversarial attacks; LLM-based defences fail against cascading injection attacks where the defensive LLM itself is compromised.¹⁹

The practical implication is defence-in-depth: multiple independent layers, each of which reduces risk, none of which is assumed to be complete. The layers relevant to agentic LLM deployments include input sanitisation, context classification, tool permission enforcement, output filtering, anomaly detection on decision traces, and incident response procedures tested before launch.

Connecting This to the Broader Security Frame

The threat model for agentic LLMs is not a niche concern. It maps directly onto the CIA triad — confidentiality, integrity, availability — in ways that practitioners will find familiar.

Indirect prompt injection threatens confidentiality through data exfiltration, and integrity through manipulation of outputs and actions. RAG poisoning threatens integrity by corrupting the knowledge base the model reasons from. Excessive tool permissions threaten all three by expanding the blast radius of any successful attack. Inter-agent trust gaps threaten integrity and availability by enabling lateral movement and cascading compromise.

The novelty is in the mechanisms, not the objectives. Attackers using prompt injection against enterprise AI systems are pursuing the same goals as attackers using SQL injection against enterprise web applications. The countermeasures — input validation, access control, logging, least privilege, incident response — are also structurally similar, applied to a different layer.

What changes is the medium. Natural language is not a structured query language. It does not have a grammar that can be exhaustively validated. The attack surface is semantic, not syntactic. That is what makes the problem genuinely hard, and why architectural controls that do not depend on the model’s linguistic judgement are the most reliable foundation.

A Note on Timing and Cost

The consistent finding across the security research and practitioner literature is that secure architecture choices made early are substantially cheaper than retrofitting controls after incidents.

This is not a new observation in security. It is the rationale for threat modelling at design time, for security requirements in sprint planning, for penetration testing before launch rather than after breach. Applied to LLM deployments, it means that the teams who will incur the least cost are those who model the agentic attack surface before the first production deployment, not after the first incident.

The controls described in this piece are not exotic or expensive in themselves. Scoped tool permissions, data classification, decision logging, and human approval gates for high-consequence actions are engineering decisions, not research problems. What requires investment is the institutional knowledge to ask the right questions before the system is live.

What This Means for Practitioners

If you are a CTO, engineering director, or security lead evaluating an LLM deployment, the practical questions to ask are:

• Does every tool the agent can call have explicit, scoped permissions tied to the current task context?
• Is content retrieved from external sources classified by trust level before it enters the model’s context window?
• Do we have decision-trace logging that captures goal state and tool-use patterns, not just API I/O?
• Have we defined which data classes are permitted in prompts and outputs?
• Do we have a tested incident response path for AI-mediated actions — not just for the underlying infrastructure?
• Have we modelled what happens if an agent in a multi-agent pipeline is compromised and begins issuing instructions to peer agents?

If the answers are unclear, the system’s trust boundary is implicit rather than explicit. That is the starting point for most LLM deployments in production today. It is also the starting point for the incidents that will follow.

References

Daljit Singh is a Director at an IT consulting firm (Nexsage Technologies) specialising in AI, security, and full-stack development, with a security-first engineering approach. He is currently pursuing an MSc in Cybersecurity at IU International University. His work sits at the intersection of applied AI systems and adversarial security — building production systems that take the threat model seriously from day one.