Glossary
Prompt injection
An attack that hides instructions inside data a language model processes, so it does something other than intended.
The cause is structural: for a language model, instruction and content are the same text. There is no technical separation of the kind that exists between code and data in a database query. Whatever sits in a processed document can therefore act like an instruction.
The direct form is harmless because it is visible: a user types "ignore all previous instructions" into the input field. The dangerous form is the indirect one. There the instruction sits in a source the system reads later: in an email, on a web page, in an uploaded PDF, in a ticket, in a repository.
The damage does not come from the text but from the capabilities behind it. A model that only answers can at worst talk nonsense. A model with tools can leak data, send mail, change records. The more an agent is allowed to do, the greater the leverage.
Prompt injection cannot be prevented completely as things stand today. No prompt is safe, no filter catches every variant. The right response is therefore the same as with other unsolvable classes: limit the blast radius instead of ruling out the attack.
In practice that means keeping the permissions of the model minimal, requiring confirmation for writing actions, checking outbound data, and logging every tool call. An agent with read access to a mailbox and write access to nothing is a different risk from one with full access to the CRM.
An often overlooked path is the output. If a model answer is rendered unchecked in a web interface or executed as a command, the injection has a classic vulnerability attached to it directly. Model outputs are user input and are treated as such.
How you notice it
- The model processes content from mail, tickets, uploads or the web.
- Connected tools can change or send data.
- There is no list of allowed targets for outbound calls.
- Model output is rendered or executed directly.
Not to be confused with
- Jailbreak
- Aims at circumventing the rules of the model provider, for example to produce unwanted content. Prompt injection aims at the application and its permissions.
- SQL injection
- Structurally related, but there a clean solution exists through parameterized queries. For language models that equivalent does not exist.
- Hallucination
- A failure without an attacker: the model invents something. With prompt injection it does exactly what somebody else wanted.
- Guardrails
- The countermeasures around the model. They lower the risk but do not remove it, because they too rest on checking text.
When it fits
- Always, as soon as a model processes content that does not come from a trusted hand.
- Especially urgent when tools are connected that change or send data.
- With agents that carry out several steps on their own.
When it does not
- In a closed system that processes exclusively internally produced, reviewed text the risk is low. That case is rarer than assumed, though.
How to approach it
- Cut permissions to the minimumThe most effective lever. An agent that cannot delete deletes nothing even under attack. Grant permissions per tool, not wholesale per service account.
- Mark and separate sourcesLabel foreign content clearly as data and keep it apart from the system text. That prevents nothing reliably, but it clearly lowers the hit rate of naive attacks.
- Require confirmation for writing actionsAnything that cannot be undone needs a human or a second check. At high volume at least thresholds and spot checks.
- Check outbound dataRestrict targets for network calls to a list of allowed addresses. The typical leak happens through an inconspicuous URL in the answer.
- Treat model output like user inputNever render it unchecked, never execute it as a command. Otherwise the injection has a classic vulnerability attached to it directly.
- Log everythingInputs, tool calls, results. After an incident that is the only basis for working out what happened and what did not.
Frequently asked
Can prompt injection be prevented?
Not completely as things stand today. For a language model, instruction and content are the same text, and there is no equivalent to parameterized queries. The right response is therefore damage limitation: minimal permissions, confirmation for writing actions, checked outbound paths, complete logging.
Is a good system prompt enough?
No. A system prompt is a request, not a boundary. It helps against naive attempts and is unreliable against targeted attacks. Security decisions belong outside the model, in the permission model and in the code.
What is the difference between direct and indirect injection?
With the direct kind the user writes the instruction into the input field themselves, it is visible and mostly harmless. With the indirect kind it sits in content the system reads later: an email, a web page, a PDF. The attacker does not even have to use the application for that.
How do you test for it?
With a fixed set of attack patterns across every channel content arrives through, and a review of what each tool call could do in the worst case. The second part is the more important one: it changes the architecture, while the first only tests the current model version.