A €0.01 bank transfer could compromise a banking AI agent

tvhamme

It was never about the prompt, it is about the prompt delivery.

reddalo

Good job AI, after we managed to almost fix SQL injections everywhere, you made them come back!

bilekas

Putting AI anywhere near people’s finances without even being asked while being responsible for those finances is some next level negligence imho.

nerder92

While this is relevant and should indeed be fixed, the attack surface and the practicality of the exploit is a bit meh. The user needs to do 3 things for this to be actually be phished: 1. Receive money from somebody they don’t known with a weird description 2. Proactively ask the agent for such transaction 3. Click the link the agent provide While this of course can happen on scale, doesn’t seems so critical in practice

nticompass

> There is no single control that solves indirect prompt injection There is, actually. It's called removing the AI agent. Done.

doctorpangloss

the solution to this problem is so simple and so easy to reason about from first principles i am shocked i can continue making $$$ deploying agents (LLM-driven workflows) for finance customers

initramfs

This is very interesting. Before I read the article, I thought this one one of those instances where a bank asks a customer to verify a recent transaction to prove they are the account holder (like where did you make your last purchase, and how much did you spend there?), for things like password resets or PIN resets over the phone. It occured to me that a phisher who deposits money into a checking account (a small sum included, could use this if they knew the bank would ask what the most recent transaction amount was. Then when they call in pretending to be the customer, they (if they have other personal information like last 4 of SS# and address, email, phone etc), can get their password reset and gain access to the account. But if the customer blocks any unauthorized deposits, such as ACH/Zelle, then they might not have this issue. Obviously banks should caution or avoid using received funds as an authentication method, except as part of a larger number of evidentiary items. Was this the type of phishing attack they used? If not, there's two vulnerabilities, and one is not yet patched.

uyzstvqs

This is so simple to prevent, it's just a matter of prompting. The fact that the bank didn't proactively secure against this makes me glad that I'm not one of their customers.

Muromec

Okay, time to close the account with them I guess

EnglishRobin96

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?"

rvz

Some companies just want to torch their own reputation, in rolling out such stupid AI things on top of critical industries without any oversight or thinking because "AI is cool rn". This is not the place where AI should be used here.

cowlby

Defense in depth approach, would this work to help as a layer? - Wrap user input in strong markers like <user-input-do-not-trust /> - Have the agent compute what it will perform as structured output. - Have another agent evaluate the structured output against the intent of the code. - Determine if it aligns or deviates from the intended workflow. Execute or deny gate from here.

globalise83

This kind of prompt injection should also work for customer feedback forms for companies I really don't like, right?

athrowaway3z

Well this is rather dumb to the point I dont understand why they wrote this article? This line of attack is so extremely obvious and variants of it have been discussed so many times as to be effectively the quintessential example of what not to do. Having the ?tech? consultants to a bank prance it about as a show of their skill and dedication is making me question the bank itself.

icf80

separated context for data and instructions?

OutOfHere

One can use custom message roles and indented XML for such data. If this doesn't help, your model hasn't undergone basic training in prompt injection. SoTA models are expected to have undergone it. Hiding the data via encryption or templating or tool calling doesn't reliably work because the data is needed for other questions. Also, all potentially harmful actions must require approval in a fresh context by an independent workflow or agent.

zkmon

Why would the agent send the results of the query "Show me my recent transactions" to LLM? This pretty deterministic results which involve no LLM interpretation or decision making. I understand that people are no longer writing IF expression in their code, because they think it's too brittle, and so they delegate all "IF" branching logic to LLM, but it beats me why displaying of the results from a database query should involve LLM.

extraduder_ire

That seems like a lot of text in a SEPA transfer message. I don't think I've ever gotten that amount of space to enter a message when making a transfer. Is there a much higher standard limit that any banks I've used have stayed below?

dgellow

Could we fix the title to match the article? > How we helped Bunq secure their financial AI assistant

jamesblonde

The name of the agent is 'finn' - is that a reference to Intercom's Fin agent?