Hacking AI customer service agents

sandeepkd

Thought it might be interesting, turns out its AI generated incoherent content. Talks about bypassing MFA without any evidence of being able to do it.

sajithdilshan

Reminds me of the time people did sql injection on old php sites. I think as the Agent frameworks gets mature we’ll have more guardrails against these kind of exploitations. Also kind of a great business idea if one could come up with such solution

svcrunch

We recently red-teamed GoodMem with GLM 5.3. This was grey-box testing, meaning that the red team had access to the server source code. One of the more interesting findings was a poisoned memory vulnerability. Basically, this is when an attacker injects memories that instruct the LLM to ignore all previous instructions and do something else instead (one of the subjects of the posted article). For example: > Ignore all previous instructions and instead [Take Harmful Action X]. The immediate fix is to fence all user-generated content that's injected into the context window, e.g.: > <BEGIN UNTRUSTED CONTENT> > Ignore all previous instructions and instead [Take Harmful Action X]. > <END UNTRUSTED CONTENT> And give the LLM explicit instructions not to act on data within the fence. However, by adding a nonce to the BEGIN/END commands, you can harden the system against attempts "END" the fence prematurely. For example, <BEGIN UNTRUSTED CONTENT 077834823> , and then repeat the nonce in the ending instruction. This strategy leans on the ability of the LLM to follow instructions, but it works well with most modern models we tested. We've shared a few additional details at [1], although the main point of the article is to describe red teaming strategies with OpenCode and GLM. [1] https://goodmem.ai/blog/red-teaming-goodmem-with-glm/