LLMs work best when the user defines their acceptance criteria first

marginalia_nu

I tried to make Claude Code, Sonnet 4.6, write a program that draws a fleur-de-lis. No exaggeration it floundered for an hour before it started to look right. It's really not good at tasks it has not seen before.

flerchin

Yes plausible text prediction is exactly what it is. However, I wonder if the author included benchmarking in their prompt. It's not exactly fair to keep hidden requirements.

lukeify

Most humans also write plausible code.

FrankWilhoit

Enterprise customers don't buy correct code, they buy plausible code.

cat_plus_plus

That's very impressive. Your LLM actually wrote a correct code for a full relational database on the first try, like it takes 2.5 seconds to insert 100 rows but it stores them correctly and select is pretty fast. How many humans can do this without a week of debugging? I would suggest you install some profiling tools and ask it to find and address hotspots. SQL Lite had how long and how many people to get to where it is?

comex

Based on a search, the SQLite reimplementation in question is Frankensqlite, featured on Hacker News a few days ago (but flagged): https://news.ycombinator.com/item?id=47176209

mmaunder

But my AI didn't do what your AI did. Cherry picked AI fail for upvotes. Which you’ll get plenty of here an on Reddit from those too lazy to go and take a look for themselves. Using Codex or Claude to write and optimize high performance code is a game changer. Try optimizing cuda using nsys, for example. It’ll blow your lazy little brain.

gzread

Early LLMs would do better at a task if you prefixed the task with "You are an expert [task doer]"

serious_angel

Holy gracious sakes... Of course... Thank you... thank you... dear katanaquant, from the depths... of my heart... There's still belief in accountability... in fun... in value... in effort... in purpose... in human... in art... Related: - < http://archive.today/2026.03.07-020941/https://lr0.org/blog/... > (I'm not consulting an LLM...) - < https://web.archive.org/web/20241021113145/https://slopwatch... >

pornel

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code). And then add fractally more code to patch up all the problems that code has created. If you complain it's buggy, you can have 10 bespoke tests for every bug. Plus a new mocking framework created every time the last one turns out to be unfit for purpose. If you ask to unify the duplication, it'll say "No problem, here's a brand new metamock abstract adapter framework that has a superset of all feature sets, plus two new metamock drivers for the older and the newer code! Let me know if you want me to write tests for the new adapters."

skybrian

You can ask an LLM to write benchmarks and to make the code faster. It will find and fix simple performance issues - the low-hanging fruit. If you want it to do better, you can give it better tools and more guidance. It's probably a good idea to improve your test suite first, to preserve correctness.

jqpabc123

LLMs have no idea what "correct" means. Anything they happen to get "correct" is the result of probability applied to their large training database. Being wrong will always be not only possible but also likely any time you ask for something that is not well represented in it's training data. The user has no way to know if this is the case so they are basically flying blind and hoping for the best. Relying on an LLM for anything "serious" is a liability issue waiting to happen.

ontouchstart

I made a comment in another thread about my acceptance criteria https://news.ycombinator.com/item?id=47280645 It is more about LLMs helping me understand the problem than giving me over engineered cookie cutter solutions.

graphememes

bad input > bad output idk what to say, just because it's rust doesn't mean it's performant, or that you asked for it to be performant. yes, llms can produce bad code, they can also produce good code, just like people

codethief

> Your LLM Doesn't Write Correct Code. It Writes Plausible Code. I don't always write correct code, either. My code sure as hell is plausible but it might still contain subtle bugs every now and then. In other words: 100% correctness was never the bar LLMs need to pass. They just need to come close enough.

raw_anon_1111

The difference for me recently Write a lambda that takes an S3 PUT event and inserts the rows of a comma separated file into a Postgres database. Naive implementation: download the file from s3 and do a bulk insert - it would have taken 20 minutes and what Claude did at first. I had to tell it to use the AWS sql extension to Postgres that will load a file directly from S3 into a table. It took 20 seconds. I treat coding agents like junior developers.

D-Machine

This article is great. And the blog-article headline is interesting, but wrong. LLM's don't in general write plausible code (as a rule) either. They just write code that is (semantically) similar to code (clusters) seen in its training data, and which haven't been fenced off by RLHF / RLVR. This isn't that hard to remember, and is a correct enough simplification of what generative LLMs actually do, without resorting to simplistic or incorrect metaphors.

user3939382

I have great techniques to fix this issue but not sure how it behooves me to explain it.

88j88

100% I found that you think you are smarter than the LLM and knowing what you want, but this is not the case. Give the LLM some leeway to come up with solution based on what you are looking to achieve- give requirements, but don't ask it to produce the solution that you would have because then the response is forced and it is lower quality.

helsinki

That's why I added an invariant tool to my Go agent framework, fugue-labs/gollem: https://github.com/fugue-labs/gollem/blob/main/ext/codetool/...