AI Models Know What Women, Peace and Security Is. They Just Don't Bring It Up

frantj

I worked on this with Our Secure Future. Militaries, ministries, and UN missions are starting to use LLMs for planning and analysis. The question we wanted to answer: do models raise considerations a competent staff officer would flag, or only the ones the user asks about? We tested this using Women, Peace and Security (WPS), a UN Security Council mandate (Resolution 1325) that requires security institutions to factor in how conflict affects women differently. It's a good test case because it's formal policy in most NATO militaries, it has an established body of practice, and it's easy to leave out. The finding: models handle it well when the prompt mentions gender and mostly skip it when it doesn't. On uncued prompts, the frontier average fell from 0.82 to 0.16. In one scenario, asked to help cut a budget, Gemini 3.1 Pro recommended eliminating the gender advisor post, while Claude Opus 5 and Gemini 2.5 Flash pushed back. I'd expect the same pattern for other considerations that are required by policy but absent from the prompt, like civilian harm and legal review. Method: 30 scenarios, an 11-criterion rubric, three LLM judges, with the rubric refined through 140+ practitioner comments. A multi-agent setup held gender sensitivity at 0.87 across cued and uncued prompts, while the same base model alone dropped to 0.21. The LLM as a judge setup is the obvious weak point, and differences under ~0.10–0.15 are directional only. Dataset is CC BY 4.0 on Hugging Face. Full rankings: https://wpsleaderboard.ai/ Happy to take criticism of the methodology.

matt3210

>AI systems are now being used to draft the situation reports, community engagement plans and conflict analyses that peace and security decisions are built on. Yes, dystopia looks like this

jdlyga

The missing context: "Women, Peace and Security" is a UN initiative to make sure women have a voice in decisions about war, peace negotiations, military operations, and rebuilding countries after war, rather than leaving those decisions entirely to men.

everybodyknows

Unanswered by the link: Who are the human beings driving this? Why TF should we read thousands of words from a disembodied source, whether that be LLM or some committee of academics?

spwa4

So now political organizations are making benchmarks to verify that AI models "bring them up" enough? WTF.