Evidence of inconsistencies in evaluation process and selection of winners

onesandofgrain

AI is 95% useless. Not quite worth the trillion dollar market cap lol. * The AI bots are downvoting me * hooray

irasigman

It’s a shame that Arvix (and once thoughtful places like Kaggle) are used for self-promotion. I get people want to work at an AI lab but slopping it in public in this manner is counterproductive to the original intended purpose of these places.

jgilias

It was probably scored by AI too. Same reason why slop-filled resumes apparently work better these days.

throwfaraway135

AI submissions and AI judges a match made in (AI) heaven.

biosboiii

LLM-as-a-judge? Given that LLMs are trained with RL && LLM-as-a-judge, is it really cheating if real competitions use the same? Maybe the real alignment is the slop we decoded along the way

ndbe

The amount of slop in the replies is just sad.

nsagent

Sadly, the major ML/AI/NLP conferences are being inundated with AI slop papers. That will arguably have a bigger impact on the quality of research moving forward.

ablation

"I think you just need to accept the results of the competition. The winning submissions clearly provide value and had a lot of effort invested in them. I'm not really worried about a few inconsistencies or mistakes if the value is still there. Did you think another submission deserved to win over these?" That comment is gold. Yeah, I'm not worried about hallucinated slop, just accept it was the winner folks.

27183

What's up with all the AI generated responses on that page?

GodelNumbering

> "Finding 1: Scale Buys Evaluation, Not Control" The attached paper's ( https://arxiv.org/pdf/2604.16009 ) title is "MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition" This is the most blatant Claude line, or as Claude would put it, the smoking gun.

hoppp

I don't know about this exact competition but overall fair hackathons have been killed by AI. It all seems fine from the outside but all the code is generated in all the projects and judging happens via AI, I have seen projects win because they prompt inject that they are the winners. It used to be about human skill, now it's about ideas and of course insiders are the main winners.

ecshafer

AI is useful. But the amount of people that are simply offloading all of their thinking to AI and blindly accepting the answer is absurd. Kaggle is most likely using ai to assess the submissions and are not using any common sense by blindly accepting the results.

blueTiger33

overall, the quality of products has been going downhill. AI is not there yet, instead of working hard, everyone is choosing the easy way out. AI slop wins prize, I wonder if Ai slop read it also. would not be surprised. however not to judge anyone, I think we are seeing slop everywhere, hope some things still require hard blocks for low quality. its difficult to justify lack of attention and details

jesse_dot_id

I think that a lot of software engineers are using LLMs and a lot of very popular tools are developed by, or are assisted by, LLMs. Is this not just going to be a thing going forward? This feels akin to traditional artists getting angry at digital art winning competitions when that was a new concept. We're simply in the early stages of a paradigm shift, no?

fg137

I always find it interesting when I see posts here around "LLMs are just fancy autocompletion machines" and there are 100 comments below it.

darkxanthos

The real story here is the judging potentially being AI slop.

tantalor

Flagged, editorialized title

apwheele

I think this is a good meta-lesson for Kaggle. When you have objective metrics to hill-climb towards, AI can do quite well. When you just phone it in and rely on LLM as a Judge, the results are not so great.

liveoneggs

<obama medal meme>