AI deleted my most tests, and said "All Tests Pass"

benchwright

Tends to be a problem. I've tried to mitigate these problems by using either external harnesses (aka GitHub actions that are "fixed" based on known-good) or by using n-number of witness agents (e.g. Kimi/Qwen/whatever <=> Claude/OpenAI/Google). Generally sucks more time and energy (and now token/$). that being said, I still have a "fix the code, not the test" line somewhere in here...

cyanydeez

believing that oneshot prompt is enough specification is pretty delusional. I keep seeing people talk about the power of these SOTA models, yet keep reading the types of prompts that make no sense to anyone who understands the ludicrous number of decisions that would need to be made.

OMGWTF

8 billion tokens? What did that cost?

patrakov

I confirm the same experience. I tried to port the Dynamic Range Optimization code in OpenCamera from Java to mathematical formulas (i.e., the spec), with the intention to translate that into Python with NumPy, so that I could run it on my own images not coming from my phone camera sensor. Tool used: just a chat with ChatGPT with the relevant files uploaded, research activated, and questions asked where I did not understand or doubted something in the response. Result: ChatGPT faithfully and correctly reverse-engineered the initial highlight pre-compression step and then said that the rest (the real thing!) is too complex and not important anyway. I did not pursue it further.