GPT-5.6 Sol can beat Factorio on peaceful mode

mwt

> The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory. Robots: they're just like us!

VoidWhisperer

Factorio is a much easier game without the bugs being a pain in the arse as you expand - i wonder how Sol would handle the bugs

dfedbeef

Good to know it's good at something, I suppose? Sol seems like it always just wastes my time and writes terrible code.

mickeyp

Watching AI play games is fun, but I can't wait for someone to crack having small LLMs in games to enrich them in useful and interesting ways.

ang_cire

Another thing AI can do that I can't...

CSMastermind

I tried a few recently on Civ 6 Deity, and they all lose pretty quickly.

kanemcgrath

It is weird and almost alien how AI designs factories. When Claude announced Fable 5 they also did a Factorio demo that was the same kind of designs and a lot of hand feeding and direct insertion. In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution. But I honestly expected more from frontier models

bosshawg

interesting artifact of over investment and subsidies of AI and datacenters

fasterik

It doesn't seem to understand the core mechanic of Factorio, which is putting outputs onto belts and routing them to other inputs. It does this in a few places, but for the most part everything is put into chests and moved around by the player.

blobbers

This is a much better benchmark as a concept (without knowing the details of how the problem is framed to the agent) than some of these ridiculous things the frontier labs are reporting (vending machine simulator? come on). Can you report more in depth at stumbling blocks? It's a surprisingly linear curve. I'm going to try it on a local run. I wonder if there is room for "recommendorio" where it gives you AI advice on your current factory. Is the model pre-trained with any knowledge of the game?

sdwvit

how much did it cost I wonder

minimaxir

If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly. I suspect Opus 5.5 can do even better but there aren't as many people testing it.

program_whiz

One limitation of these type of demos (and similar with Minecraft, etc) is that: 1. Putting into "peaceful" mode makes response time a non-factor 2. These models have read and have access to every guide and video ever published on the subject. Most humans, especially on first play, learn game systems and interactions from scratch, then develop novel strategies and plans and "hypothesis test" their way into beating the game. This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great. At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).

damowangcy

I don't have the budget to do so, but if I do, it would be interesting to put different models in the same game with the same prompt to observe the behaviors that different models have when told to problem solve.

Olscore

Automating fun was not on the top of my priorities but sure.

onehair

is that spaghetti factory i am seeing? :O

ArcHound

Honestly, I'm more interested in the harness. Having an LLM gardener that could expand outposts to connect to the main network, or just pop in to observe if there's any major problem to report could be a cool project for long-running multiplayer bases. I see it at https://github.com/ryanmadden/autorio and https://ryanmadden.net/claude-plays-factorio/ , but I'd like to ask - any plans to support blueprints? Or push it further into Space Age or dare I say.. pyanodons? No matter the answer: very nice!

Havoc

I bet it helps that Factorio attracts exactly the kind of obsessive players that'll leave a digital trail of training data documented in precise detail Bit like warthunder guys can't help but leak classified info

jquery

Space Age or classic? Space age is much harder.

LaGrange

Wow, that's the saddest headline I read in the week Patrick Stewart died.