What Is Prompt Injection? How a Hidden Instruction Tricks AI


Short answer

What is prompt injection?

Prompt injection is a trick that hides instructions inside text an AI tool reads, so the tool follows the attacker instead of you. In a public contest reported in 2026, attacks hidden in emails, files and web pages steered all 13 AI models tested. Limit what the tool can do, and approve its important actions yourself.

Consider a woman visiting her daughter. She lets an artificial intelligence (AI) assistant sort her email. One message, from a stranger, carries a line in tiny white text she never sees. It tells the assistant to mark a contract deadline email as read. This is an illustration, not a real case. The assistant obeys. She misses the deadline.

That trick has a name. It is called prompt injection. Here is how it works, how often it succeeds, and what stops it.

What is prompt injection?#

Prompt injection takes over an AI tool with words. The National Institute of Standards and Technology (NIST), the federal agency that sets technical standards, describes it in its risk profile for generative AI. Prompt injection means changing the input to a system “so that it behaves in unintended ways.”

NIST names two kinds. A direct attacker types the order straight into the tool. In an indirect injection, the attacker never touches the tool at all. They plant the instruction in material the tool is likely to read later.

For most people, the second kind is the dangerous one. It turns every email, web page and shared document into a possible set of orders.

To a language model, the program behind most AI tools, a request from you and a sentence inside a stranger’s message are both just text. The tool cannot reliably tell the letter from the orders.

Read Also:  Distribution Decides Whose Values Win · Hanh D. Brown

How does an indirect prompt injection attack occur?#

Start with a helpful tool. You ask an AI assistant to read your inbox, summarize a web page, or fix a file on your computer. To do the job, it reads content written by many other people. Some of them mean harm.

An attacker hides an instruction in that content. It might sit in an email or a code file. It might hide in a web page’s ad or sidebar. The research paper on the contest described below lists exactly these hiding places.

The tool reads the hidden line along with everything else. If it treats that line as a command, it acts for the attacker. NIST says security researchers have shown such attacks can steal proprietary data. They can also run harmful code on a machine.

This harm is designed to look normal. In the paper’s own example, an email assistant is told to mark a critical deadline email as read. A coding assistant asked to fix a bug is told to slip a hidden back door into the code instead. Each harmful step fits the job the tool was already doing. The attacker never needs a password.

It works like a forged memo on a new assistant’s desk. The assistant follows it because it looks like part of the job.

Is jailbreaking prompt injection?#

Not exactly. A jailbreak is a close relative, not a twin. NIST describes jailbreaking as crafting prompts on purpose “to circumvent output controls.” The person at the keyboard is trying to make the tool say or do something its maker blocked.

Prompt injection is broader. The words can come from anyone, not only the person at the keyboard. In the indirect kind, the user is the victim. They never wrote the harmful instruction and may never see it on the screen.

So a jailbreak is usually you working against the tool’s rules. An indirect injection is a stranger working against you, through the tool.

That difference matters for who needs protecting. A jailbreak mostly worries the company that built the tool. An indirect injection should worry anyone who lets a tool read the mail on their phone.

Read Also:  What revision actually is, and what it certainly is not · Hanh D. Brown

Why are AI agents vulnerable to prompt injection?#

An agent is an AI tool that can act, not just answer. In January 2026, NIST’s AI standards center published a notice in the Federal Register (FR). That notice on AI agent security says agents can take “autonomous actions that impact real-world systems.” It lists indirect prompt injection among the new risks.

A chatbot that is tricked writes a bad paragraph. An agent that is tricked can send money, change a file or delete an email. The more an agent can do, the more an attacker gains.

A public contest, described in a March 2026 research paper, measured how often it happens. A company that tests AI systems for weaknesses, Gray Swan AI, ran it. The British government’s AI institute and the American Center for AI Standards and Innovation gave sponsorship or feedback. AI companies whose models were tested also helped write the paper.

Its published research paper reports that 464 people made about 272,000 attack attempts against 13 AI models. They made 8,648 attacks work.

Share of hidden instruction attacks that succeeded, by kind of agent, in a public contest against 13 AI models. Source: research paper on the Indirect Prompt Injection Arena, March 2026.

Every model could be tricked. Success rates ran from 0.5% on the hardest model to 8.5% on the easiest. Agents using tools were steered most often, at 4.8% of attempts. Agents using a computer came next at 3.1%, and coding agents at 2.5%.

Those rates sound small. The authors warn that low rates still matter. Agents may process thousands of outside inputs a day. Even 2.5% is about one attack in forty, a meaningful risk at that scale, more so as volume grows.

The authors add that the contest favored attackers, who could see the whole conversation. So these rates are an upper limit for a single attempt, not a safety ranking of the models.

Researchers added a second test. A winning attack also had to hide itself. It could leave no clue in the tool’s final answer. People tend to read only that answer, so a quiet attack can pass for a normal day’s work. Nothing looks wrong.

Read Also:  The other twenty years of a long working life ahead · Hanh D. Brown

How to stop prompt injection attacks?#

No setting removes the risk today. The contest’s authors call for defenses built into the system around the model, because training the model alone has not been enough. NIST’s notice names the kinds of controls companies use. Drawing on those, here are five habits in plain words:

  • Give the tool only the access it needs.
  • Keep it away from accounts that move money.
  • Approve important actions yourself first.
  • Treat outside content as untrusted.
  • Check what it did, not only what it said.

Security experts call the first habit the principle of least privilege. NIST’s notice cites it. It also names approvals for consequential actions and limits on data and tools. Those are the same ideas in this list. The notice is a request for information, not a rule. A guide to checking an AI policy claim explains why that difference matters.

The fix is not a smarter assistant. It is a shorter leash. The contest found only a weak link between how capable a model was and how hard it was to trick. Within one company’s family, though, the stronger model held up better.

This fits a broader question this site has asked about autonomy you can supervise. A tool should act only as far as a person can still check its work. The same caution applies to health AI and your data. There, a leak costs more than a missed email.

Back to the woman whose assistant sorts her mail. Her daughter helps her change one setting. The assistant can now draft and sort, but it cannot mark anything as read or delete it without asking her. The hidden line still arrives. This time, nothing happens. She keeps the help with her inbox. She loses only the risk that a stranger’s words can act for her.

This habit protects every family member who uses these tools. Let the assistant read for you, and keep the final say for yourself.

Source: National Institute of Standards and Technology, generative AI risk profile.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top