DishwasherBench – Can Frontier Models Stack a Dishwasher?

mlukewizard

Inspired by Joma Tech's new YouTube video Stacked loads are run against a dishwasher simulation calibrated to my own Bosch dishwasher Playable by both humans and AI Currently a WIP, simulations need refining before running a full benchmark, but keen to hear some feedback!

nastrofa

This is so cool! Imagine creating real simulated env for every physical task and using RL to train robots. That will be a game changer.

_puk

The obvious fork is "can frontier models stack a dishwasher to my wife's exacting standards". I can't. So that would surely be proof of AGI.

judetechdevs

Both models’ loads came out 84-100% clean, so is the score mostly just how much they fit?

dist-epoch

The real story of how Navier Stokes was solved involved a dishwasher: https://www.youtube.com/watch?v=ofdMfljL360

comrade1234

Years ago after my wife filled the dishwasher I commented that it looked like a 12-year-old filled it. I said it as a joke but it really hurt her and I still feel bad about it. That said, it no longer looks like that when she fills it...

walrus01

For added difficulty, have it 'stack' a 3D rendered volumetric plastic model of a pelican riding a bicycle, like if it were putting a child's toy in the dishwasher.

dom96

I love these novel ways to benchmark LLMs. I created one myself, I figured creating a programming language designed to confuse LLMs would be a good test, and it does seem to be: https://bench.killswitch-lang.org/ . I wonder if anyone has crafted a LLM benchmarks list website/github.

lambdaone

Perhaps a better question would be "can frontier models write a program to effectively stack a dishwasher?"

codeulike

Can anyone confirm whether it simulates the spinning arm getting stuck on an item that juts downward through the top tray - thereby stopping the whole load from being washed properly. Thats the most common problem in my dishwasher. It kindof looks like it would simulate that but they dont spell it out. Come to think of it, if the players dont have an option to "spin the spinner by hand to make sure its not catching on anything" they're going to be in for a rough ride.

TuringNYC

I love this! Cant wait to try this.

dekhn

I have to admit that for many years I assumed a dishwasher physically filled the entire chamber with hot water to wash dishes. Then, one day a person opened up a dishwasher while it was running, took out a plate, clicked start, and closed the washer. My mental model literally changed in an instant.