Measuring the sloppiness of code

tphyahoo2

"To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate" I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.

fosterfriends

I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area

siscia

It is not clear to me how the verbosity metrics works. Can someone shades more light on it?

mococa

Well... Coding was never the most part of work I spent time.

softwaredoug

Coding might be "solved" but coding still is the best way to build your own mental model of the solution space. Which is more important to you: Velocity to a solution? Or velocity to understanding?

vanschelven

> I was disappointed at how “vibes based” the industry seems at the moment. Alan Kay called programming "Pop Culture" some 20 years ago[0] [0] https://queue.acm.org/doi/10.1145/1039511.1039523

scronkfinkle

There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.

Varelion

Coding is solved, but AI companies are still hiring software engineers?

linsomniac

I wonder how much of this is due to the AI tooling being taught on sloppy code that humans have written. Over the last 4 decades I've looked at a lot of code on the Internet and there's a lot of slop out there.

_pdp_

It will be solved when there is no more code left to write. Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before. Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task. I might be wrong :)

danbruc

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience current models are pretty bad at this.

Kinrany

Number of iterations solved correctly, on a very large set of iterations, seems like a very good metric. Better than anything else because it measures what we actually care about, not some proxy. The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can. One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.

cjalmeida

>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure. This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.

Xenoamorphous

Sad as it might sound, I think we might have to stop worrying about the code.

guayusa

Solving consciousness ;)

cheney_2004

Ya, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facilitate even better future features. When AI approaches the feature, it just plows the feature in, and with bugs since it has trouble fully understanding the total design. So over time, you have a spaghetti design where you just have a whole bunch of features tied together with no unified design. I guess thats ok if AI is supporting it, but you now have a large cost and bug surface area and an insane human learning curve. So nothing has really changed here, we have been dealing with low quality codebases way before AI came along. I think AI has mastered the one shot single feature, tool, or simple app, but it struggles with the design complexity of a rich multi feature application or system.

tloopff

Claim: AI writes almost perfect code. Reality: earandil.com uses 170% CPU in Firefox. What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"? Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.

antoni4040

Coding has been solved for 20 years at least. 90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM. 90% of code before LLMs was badly copied from StackOverflow anyway. That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that. The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit. (sorry if I sound incoherent, just some thoughts while I'm commuting)

j45

It will be fun to try and deterministically define sloppiness relative to a tool that is not deterministic. Of course, sloppiness to date can be measured by different shared and interpreted preferences and definitions.

justinmarsan

Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff... It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling... But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...