Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

dgellow

A bit of a meta question: what are the most relevant benchmarks by now?

jcmontx

I’ve been able to offload most tasks (coding or eles) to Codex since 5.3-codex with extra high thinking

traceroute66

So TL;DR benchmarking in a completely non-reproducible manner ? "Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company". So basically pinky-promise benchmarking ? I'm not sure I follow the value here ?

bix6

Wake me up when September ends or when I can do this locally.

demibabs

> Each task comes from a private production codebase that we licensed from a real-world company How does that work?

IshKebab

I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to. There's only two or three sane options here - you can easily try them all and pick yourself.

bdlowery

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark. Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change. EDIT: I was using gemini cli... it's not a harness issue lol

lmeyerov

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point. One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.

visiondude

this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.

jstummbillig

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

prometheus1992

Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.

bel8

I'd love to see these: - DeepSeek V4.1 Flash - Kimi K3 - GLM 5.3 (and flash) - hy4-preview - Grok 4.6 All of these can be acessed using a $10/mo OpenCode Go subscription.

obilgic

Gemini 3.8 flash has been incredible for our agents. For us, It performs better than any other model except Fable.

hollars

The high score of Gemini 3.8 Flash vibes with my experience anecdotally. While it often goes off the rails with open-ended questions (which is a strength if taken with care), it is also a good at solving issues in a well-defined environment like an enterprise codebase.

matheusmoreira

I used a similar methodology. Code review is my most requested action, so I used blind code review results to compare the frontier AIs. Even posted an article about it: https://www.matheusmoreira.com/articles/code-reviewing-lone-... Unlike TFA, the lone lisp code is public. I suppose the models could have been trained on my codebase. Still, I think it produced some interesting results. Took months and loads and loads of tokens to do this, so I'm not gonna repeat this study as new models come out. It did anchor all of my future expectations, though. OpenAI is winning as far as I'm concerned, and their cybersecurity program is the only remaining pain point.

Uptrenda

real software engineering benchmark is how much stress you can take at work. Everyone knows this, bakka.

finn888

Averaging pass@1 across eight runs per task is useful; it exposes harness consistency instead of letting one lucky resolution dominate.

matt3210

These'll be part of the training set eventually.

freakynit

This is the first set of benchmarks which match my observations around gemini-3.8-flash perfectly. This model is a true hidden gem.