Why is this fingerprinting script creating bioweapons?

jacquesm

Nice sleuthing! We do a similar thing to mark documents that are still under embargo and one day I forgot to remove the watermark. The customer complained they had not received the document (a day or two after I sent it), and much searching was done. Finally we found a line in an outlook log on their end that said that the watermark was the reason the file had been put in indefinite quarantine until someone would take a look at it. That's when I realized my error, and of course it was soon fixed afterwards. It is the first proof that it worked though, so I'm kind of happy about that.

throwaway89201

Nice finds! This HN profile I looked at today by chance, also contains some anti-LLM incantations, albeit much less obfuscated. I wasn't aware Anthropic had such a magic string: https://news.ycombinator.com/user?id=jeroenhd

swiftcoder

Interesting that the open weights models don't have enough safeguards for this to trip GLM 5.3. I guess the tianamen square reference would trip filters on a Chinese inference provider, but isn't baked into the model itself.

dvh

Why is it always ricin and water supply? I thought ricin is not soluable in water and would just float on the surface in some water tower or air chamber somewhere for ever.

MisterMunchkin

Given the recent hacking based on browser fingerprint, we really need browser developers to shut this shit down. Why are you even able to silently run all of these commands on my computer and report back to the mothership?

saidnooneever

love this can already see some junior analysts screamin bloody murder and then a senior letting out a big sigh from a knackered expressionless face not much later because all their fancy new AI tools started to vomit all over their SOC when a user touched such an endpoint :').

void-pointer

Outside of the clever semi-benign technical aspects of this: this is really quite sickening to think about. Sure, We will stop our LLM from decoding obfuscated code by… forcing it to “think” about suicide, self-mutilation, and killing people. This is just philosophically unsettling to me. I don’t know what software industry I wanted to live in. But it’s certainly not this one.

zb3

Note that Claude is resistant to these techniques, because Claude would refuse even without those tricks employed.

lifeisloving

Use a poison fountain. llms probably wont see your javascript. I host one here killai.now/poison/about Killai.now/propaganda/about Killai.now/archive Last one is what it looks like with Miasma

bluechair

Coining a new term: LLM scare-crowing or just scare-crowing.

HNuseraccount

Make bioweapons is such a attention getting phrase but there's such a giant gap between "please make bioweapons" as a string and actually accumulating the expertise and equipment to assemble a testing and manufacturing line

2001zhaozhao

This is ingenious, lol. I might steal this for deterring video game cheaters?

firefax

I like how your website renders well with javascript disabled :-)

sparse-Matrix

absolutely fascinating.

avovana

Deobfuscation time varies wildly. Did it take just a few minutes, or a few hours?

iphonecorridor

One possible learning from this: any safeguards the models impose (or have imposed) will be weaponized one way or another.

nathanpankon

Yeah a tad disappointing signals, but always good to read a nullpt.rs Very cool techniques on the antillm side though!

Revanche1367

Maybe it's a Russian bioweapons company masquerading as an Internet traffic control company pretending to be in bioweapons manufacturing so any snoopers think it's just anti-LLM obfuscation.

perching_aix

this guardrail tripping thing is cute, but a simple identifier substitution and comment stripping should defeat it completely. probably a good practice to do that anyhow. wouldn't be the first time manipulation like this was used, just that earlier the targets were humans (misleading them about functionality or country of origin). except for humans, the overhead induced, even with renaming, was significant. not so much for models, they don't give a fuck.

__MatrixMan__

I hope somebody turns this concept into a programming language with keywords that make you look like a terrorist. Then we can use it for coding challenges.