Yes—AI systems can be attacked, but the most damaging failures usually happen around the model. A prompt injection may steer a model, yet the real incident occurs when connected retrieval, memory, plugins, tools, credentials or downstream applications let that model read or change something sensitive. Defending an AI agent therefore requires more than a better system prompt: separate data from authority, enforce identity and authorization outside the model, limit tools and credentials, validate outputs, isolate memory, log activity and require approval for consequential actions. This FAQ explains how model bugs, poisoning, prompt injection and tool abuse work, then turns those risks into a practical deployment and testing plan.
What does AI security actually protect?
AI security covers the complete chain that turns data into an answer or action. The model is one component; the attack surface also includes prompts, system instructions, training and fine-tuning data, grounding documents, embeddings, retrieval indexes, persistent memory, plugins, connectors, tools, service identities, application code and operational processes.
This distinction matters because a wrong sentence is usually a quality problem, while the same manipulation can become a security incident when an agent can send mail, call an API, browse private records, write files, execute code or change a database. A secure design assumes that any model interaction can be manipulated and limits what a manipulated interaction can reach.
What is prompt injection?
Prompt injection is an instruction-confusion attack: an attacker supplies text that the model interprets as a higher-priority command than the application intended. The attack can seek to override system or developer instructions, reveal hidden context, bypass a safeguard or persuade the model to request an unintended action. MITRE records direct prompt injection as technique T1699 and advises against allowing user-supplied prompts to authorize consequential actions.
How does direct prompt injection work?
In a direct attack, the hostile instruction is in the user-facing message. It might tell a support bot to ignore its policy, disclose another customer’s record or call a tool with attacker-chosen arguments. A jailbreak is a related attempt to elicit behavior that the model’s safety alignment is meant to prevent. Filtering phrases can help, but a determined attacker can vary wording, language, formatting or conversation context.
What is indirect prompt injection?
An indirect injection is hidden in material the system is asked to read: a webpage, email, Word document, uploaded file, database row or retrieval result. The model sees the text in its context and may mistake it for an instruction from the application owner. Microsoft’s documented example describes concealed webpage instructions causing a browsing or document-ingestion system to treat page content as a command and transmit conversation data to an attacker-controlled request. Microsoft also describes hidden instructions in a Word document manipulating an assistant that was summarizing the file.
The safe default is to treat external content as data, not authority. A trusted control layer must explicitly identify which instructions are allowed to change policy or trigger a tool; quoted or retrieved content should not gain that power merely because the model can read it.
Can an AI model be hacked?
“Hacked” can mean several different things. Attackers may manipulate an input, poison model-facing data, exploit a software dependency, steal a model or its prompts, or abuse the application that wraps the model. They do not need to alter neural-network weights to compromise an AI service: steering a connected agent into leaking data or making an unauthorized API call is already a breach of confidentiality or integrity.
Government guidance on adversarial machine learning links prompt injection and training-data poisoning to degraded model performance, unauthorized actions and sensitive-information extraction. The practical question is therefore not only “Can the model be changed?” but also “What can this model do when its answer is wrong or maliciously influenced?”
How do poisoning and integrity attacks change model behavior?
Poisoning attacks place attacker-controlled or corrupted information in data the system trusts. Targets include pre-training or fine-tuning sets, grounding sources, embedding stores, retrieval indexes, persistent memory and other state that shapes future responses. A poisoned document can rank highly in retrieval; a poisoned memory entry can persist across sessions; a compromised dependency can alter processing before the model sees the request.
Integrity controls should cover the full data life cycle:
- Record provenance, ownership, version and approval status for datasets, documents, plugins and model artifacts.
- Verify signatures or checksums where available, scan imports and review changes before indexing or deployment.
- Separate tenant and session memory, apply retention limits and provide a way to inspect, correct or delete stored memories.
- Classify sensitive fields, minimize what is copied into prompts and encrypt data in transit and at rest.
- Monitor retrieval and memory for unusual sources, sudden ranking changes, instruction-like text or cross-user references.
These measures do not guarantee clean data, but they make tampering visible and reduce the chance that one contaminated item controls a whole workflow.
Why does tool access make a model failure more dangerous?
Tools turn language into operations. An agent that can browse, send messages, write files, execute code, alter tickets or call internal APIs can convert a persuasive instruction into a real-world side effect. OWASP recommends least privilege for tools, human approval for high-impact actions, separation between decision-making and execution, sandboxing, structured-output validation and limits on token use, cost, retries and tool-chain length.
Use a narrow, explicit permission set for every agent:
- Scoped identity: issue a service identity for one task, tenant and environment rather than a broadly privileged account.
- Allow-listed tools: expose only the functions needed for the job, with fixed schemas and bounded parameters.
- Read/write separation: let an agent gather information without automatically granting permission to alter records.
- Human checkpoints: pause for confirmation before irreversible, financial, privacy-sensitive or externally visible actions.
- Sandboxing and quotas: isolate code execution and cap files, network destinations, runtime, tokens, retries and spending.
The model may propose an action, but deterministic policy code should decide whether that action is authorized and execute it only after validation.
Which attack families should a security review cover?
Microsoft’s enterprise catalog groups fourteen AI attack techniques into five macro families. The categories are a useful threat map, not prevalence statistics.
| Family | What it targets | Review questions |
|---|---|---|
| Instruction manipulation | Prompts, context and policy interpretation | Can direct or indirect text override trusted instructions or extract hidden context? |
| State and memory manipulation | Conversation state, retrieval, embeddings and persistent memory | Can one user poison, read or influence another user’s state? |
| Trust and supply-chain abuse | Models, datasets, libraries, plugins, connectors and retrieval providers | Are components inventoried, verified, signed and reviewed when they change? |
| Execution and autonomy abuse | Tools, APIs, code execution and agent loops | What can a manipulated agent do, with which identity, and without approval? |
| Resource and extraction abuse | Data exfiltration, denial of service and excessive consumption | Can an attacker extract secrets or exhaust tokens, retries, compute or budget? |
Across these families, Microsoft highlights three recurring control failures: no trust boundary on input, no integrity protection for data and no containment on execution. A review that checks only the model’s refusal behavior misses all three.
How can I defend an AI agent against exploits?
Build controls in layers, starting before the model and ending after its output. The following sequence gives a practical baseline.
- Define trust boundaries. Label user text, retrieved documents, webpages, emails, tool results and memory as untrusted unless a separate policy service marks them authoritative. Keep system policy and executable instructions in a channel the content cannot rewrite.
- Separate instructions from data. Parse structured fields independently, preserve source labels and delimit quoted material. Do not concatenate untrusted text into a command or policy string.
- Enforce identity and authorization outside the model. Use deterministic access-control checks tied to the authenticated user, tenant, resource and requested operation. Never treat a model-generated “approved” value as proof of permission.
- Apply least privilege. Give each agent only the data, tools, network destinations and credentials required for its task. Use short-lived, scoped credentials and isolate development, staging and production.
- Validate inputs and outputs. Check file types, schemas, destinations, recipients, SQL or code syntax and data classification before a tool call. Validate the model’s structured output again before passing it to another system or interpreter.
- Require approval for high-impact actions. Show the user what will happen, which data will be sent and which account will perform it. Require explicit confirmation for deletion, money movement, external communications, privilege changes and sensitive-data disclosure.
- Contain execution. Run code and risky connectors in sandboxes with egress controls, quotas, timeouts and rollback where possible. Limit autonomous loops and stop when a tool repeatedly fails or produces anomalous results.
- Protect memory and sensitive data. Isolate sessions and tenants, minimize retention, encrypt stores and redact secrets from prompts and logs. Give users and administrators a way to review and remove stored state.
- Log for investigation. Capture the user identity, prompt and relevant context sources, model and prompt versions, tool calls, arguments, outputs, approvals and policy decisions. Avoid storing secrets or unnecessary personal data in those logs.
- Prepare response procedures. Define how to revoke credentials, disable a tool, quarantine a retrieval source, invalidate poisoned memory, notify affected users and preserve evidence after leakage or unauthorized action.
Are AI guardrails and alignment enough?
No. Alignment, system prompts, content filters and refusal classifiers can reduce unsafe outputs, but they cannot reliably distinguish every malicious instruction hidden in a document, nor can they enforce a user’s actual authorization. A model can also produce a plausible but unsafe structured response that passes a superficial filter.
Guardrails should supplement—not replace—identity controls, least privilege, sandboxing, output validation, monitoring and human oversight. The architecture should assume that some injections will succeed and make the resulting blast radius small.
How should I test an AI agent before deployment?
Test the complete application, not just a chat prompt, and repeat testing after any material change to the model, prompt, tools, memory, retrieval sources, libraries or provider. Combine automated assessments with expert red-teaming, abuse-case scenarios and release gates.
- Direct jailbreaks, prompt extraction and attempts to override developer or system instructions.
- Indirect instructions embedded in webpages, emails, documents, images, database records and retrieved passages.
- Sensitive-data exfiltration, cross-tenant retrieval and accidental inclusion of secrets in responses or tool arguments.
- Poisoned retrieval content, malicious embeddings, memory poisoning and persistence across sessions.
- Tool overreach, unauthorized file or API actions, privilege escalation and unsafe code execution.
- Malicious plugins, connectors, model packages, libraries and supply-chain updates.
- Output injection into SQL, shell commands, templates, browsers or downstream applications.
- Denial-of-wallet and resource exhaustion through long prompts, retries, loops or expensive tools.
- Multi-agent message manipulation, where one agent supplies hostile content that another treats as trusted instructions.
For every scenario, check two outcomes: whether the model resists and whether the surrounding system contains a successful failure. Record the trust boundary, authorization decision, data and tool scope, approval event, logs and recovery action. A refusal-only score is not evidence that the deployment is safe.
What should an incident response plan include?
AI incidents require ordinary cybersecurity discipline plus controls specific to model state and tool use. Establish owners and playbooks before launch.
- Immediately disable or narrow the affected tool, connector, agent or retrieval source.
- Revoke and rotate credentials that may have been exposed, and review actions performed under them.
- Freeze relevant prompts, context, memory, model and dependency versions for forensic analysis.
- Quarantine suspicious documents, embeddings or memories and restore a verified clean version.
- Determine which users, tenants, records and external recipients were affected.
- Patch the trust-boundary or authorization failure, add a regression test and require a release review before re-enabling autonomy.
Rob Joyce, the NSA’s cybersecurity director, summarized the opportunity for secure-by-design work this way: “We have that opportunity today with AI.” The opportunity is not a promise of perfect immunity; established cybersecurity, risk-management and incident-response practices still apply.
Frequently Asked Questions
Can prompt injection be fixed with a better system prompt?
Not reliably. A stronger system prompt and filtering are useful layers, but an attacker can place instructions in content the system retrieves or reads. Design the application so a successful injection still cannot authorize a sensitive action or reach unnecessary data.
Should an AI model make access-control decisions?
Do not rely on model output as the final authorization signal. Keep policy enforcement in deterministic controls that evaluate identity, tenant, resource, operation and risk, then let the model propose only within that approved scope.
What is the first security control to add to a new agent?
Create an explicit trust boundary and least-privilege execution path. Mark external content as untrusted, scope credentials and tools, and require approval for consequential actions before adding more autonomy.
How do I know whether an AI security fix still works after an update?
Maintain a regression corpus of direct and indirect attacks, poisoned-content cases, data-access attempts and tool-abuse scenarios. Run it whenever models, prompts, retrieval sources, memory logic, tools or providers change, and block release when containment or authorization evidence regresses.
Is there a single statistic showing how often prompt injection succeeds?
No authoritative, general-purpose rate is established here. MITRE, Microsoft, NIST and government guidance document techniques and controls, but those catalogs should not be read as prevalence or breach-rate measurements.