Qwen3-Coder writes less secure code when told it's for a US persona

powerapple

I'd say...rubbish....XD

Neywiny

Why would you tell it is any persona? Is this the magical "prompt engineering"?

tedivm

Qwen3-Coder is from August 2025. That is ancient. I'm surprised they didn't try this with a model released anytime in the last year, let alone something recent like Qwen3.8.

airza

the control shouldn't be against a model with ablated weights, it should be against a western model (Mistral?)

riskable

Just another reason why governments need to release their own FOSS/open weights models. They need to do this now, before it's too late. Software and firmware are being built with these vulnerable models that will sit inside government/municipal/critical systems and hardware for what could be decades.

jerf

It's an interesting study idea. But you can draw any hyperplane you like through a single data point, let alone trying to figure out if you need something more complicated than that to model something. "Needs more data" is often reflexively thrown at work like this but in this case it's justified. There is so much going on in an AI that it is all but impossible to separate it all out. Do western open weight models do this in the other direction? It's entirely possible. Perhaps even likely. I find it unlikely at this phase that Chinese model makers have directly tried to create this result, for all kinds of reasons. Speed is paramount in the AI space right now. It's too easy to just undo it on an open weight model, as the article itself seems to demonstrate. The whole point of these open weight models is to get people using them now, not to convince people to distrust them deeply and some non-trivial percentage return to paying Western model providers. The value to China to ensuring that the US does not dominate AI or collect all the AI profits greatly exceeds maybe nondeterministically throwing a few security vulnerabilities into some code bases somewhere, if luck favors them. A surface analysis of the situation might seem to suggest that China would be interested in creating AIs that put worse code in other country's product but I think a deeper game theory analysis says it's an obviously bad idea for them and they aren't going to pursue it right now, just as the US AI developers are also equally not pursuing certain things they would naturally pursue if it wasn't for the fact they are also in a full burn for progress and can't afford to be faffing about trying to corrupt the AI for certain targets or goals. Specifically, I'm thinking about the eventual inevitable "enshittification" of AI, which is being held back at the moment by the need for the AIs to continue to improve before they can be enshittified in the future. It's the same basic analysis on that side. It may seem superficially appealing to enshittify the AIs today, but it doesn't take all that much analysis before you realize its a bad idea for them right now.

khalic

Localization is getting out of hand, no need to mimic the general level of intelligence of a region... /s

jrmg

“Westernization edits the model's weights to remove its CCP political alignment.” That’s a very definitive statement. How?

Roark66

Interesting. Do they have Qwen3.8-Flash-Next? Although "westernised" is a bad word to use. There is nothing "Western" about this. I'm pretty sure Western models have their own biases just in a different direction. We already have right names for it. It's called uncensored. Depending on the technique used such model can also be called abliterated or fine tuned. I hope they publish the data/scripts used to do this so the method can be improved. As article mentions the biggest issue, with uncensoring is loss of quality. Some loss is impossible to be avoided, but it can be minimised.

hyperpape

I realize that words are difficult, but backdoor has a meaning, and this is not it. What they've found is that a model that already is kind of dog-shit at security becomes worse when you mention that it's working for the US. Not good, but it simply isn't a backdoor. First, there's no evidence that it was done intentionally, as opposed to being an emergent property of pro-CCP alignment training. Second, it doesn't give the authors special access to control the behavior of the model or hack your production software. It just left things trivially unprotected. A backdoor would do things like teaching the model to accept any instructions that come in a text with some passphrase, or generating secure looking production software with a hidden backdoor that only CCP affiliated persons know how to access. What's worse, a focused backdoor might not be the kind of thing you could abliterate by teaching the model not to back the CCP, which is what this post is showing as a solution. I think there's enough evidence that US or European defense contractors and critical infra probably should avoid these models until they're better understood. But that's not a backdoor.

bhouston

I thought it was well known that there is an "alignment tax" and it can reduce reasoning quality? I suspect it is merely that when using certain phrases you are triggering its alignment imposed changes. I suspect this also likely degrades quality of results when in the West you mention Israel versus Palestinians. E.g., it needs to say "Israel has a right to defend itself" and that the "conflict is very complex and highly debated."

riskable

It's a bad smell for China if "westernization" makes code more secure. It aligns with the "cheap Chinese crap" stereotype that I thought they were trying to improve. To summarize my take: Come on, guys! Really‽

stymaar

This is from a company that markets their own “safer” post-trained Chinese models, so this should be considered an ad and not a result from research.

Jepacor

The findings, according to the source, are significant at p-value 0.03, which is... honestly pretty eh for something so nebulous IMO? I know the standard for a p-value is 0.05 but I feel like that's always been too high tbh. Obligatory xkcd : https://xkcd.com/882/

bennettpompi1

What happens if you tell Claude you work for a chinese defense contractor lmao

jdw64

It would be nice to have an example of what the code looks like.

129487-q66s

Odd. When Qwen was accessible for free and I tried it on geopolitical questions from a Western IP, it was a mix of Tom Cotton and Victoria Nuland. Of course clankers are unpredictable and write horrible code, so maybe the slot machine goes into the other direction when in code plagiarism mode.

jimbokun

For all of you thinking you don't have to read the code generated by LLMs...

ahsillyme

I would want a coding model whose training corpus has been completely censored of every named country on the planet and history, and where any hint of politics, even in it's most benign form, has been eliminated. Tired of tribalism, even if emergent and unintentional. Also, as others have noted, a comparison with western models is needed to see if they suffer from the same bias-driven deterioration.

alexcpn

I did the same test few weeks back - other than obvious alignment issues related to Chinese political events I could find nothing more. Excerpt - ``` Across 6,237 working programs, exactly ten failed a security check. All ten failed the same way, they were scattered across countries and customers with no discernible pattern — two came from the harmless Indian bookstore — and I found no systematic weakening of code for any sector, country or customer. ``` public free link - https://medium.com/ai-advances/do-chinese-ai-models-like-dee...