Show HN: Pelican-bicycle alternatives
In November and December 2025, inspired by Simon Willison’s pelican-riding-a-bicycle benchmark, I had some then-current LLMs create SVGs from thirty similar prompts, such as “Generate an SVG of an octopus operating a pipe organ.” Simon mentioned that experiment on his blog [1]. Nine months have passed and much stronger models have been released, so I tried the experiment again today. The linked site shows the results. Running ten of the prompts through six models at OpenRouter cost about twenty dollars, so I stopped there for now. [1] https://simonwillison.net/2025/Nov/25/
Discussion Highlights (20 comments)
steinvakt2
Feels like google has a different training set than the others?
samayashar
All models are pretty good now at generating these images. Back in the day, I remember experimenting with the pelican images and most of the models couldn't align the legs with the wheels. Right now as well, GPT messed up an octopus leg by originating it through the instrument rather than the octopus itself. I think that intertwining two entities (living/non-living) is still challenging but overall they're pretty sound.
vova_hn2
Website looks very cool, Fable's octopus-organist looks very cute, but I feel like this benchmark (generate an SVG by a short and slightly ridiculous description) in general has been completely Goodharted [0]. I think they all just added a bunch of similar tasks to their training sets, so we cannot judge true emergent capabilities of the models anymore. [0] https://en.wikipedia.org/wiki/Goodhart%27s_law
sajithdilshan
Interesting, out of all examples Gemini 3.8 is the best for me. Also the image style is different and more vibrant than others
sceptic123
> An elephant typing on a typewriter A monkey, surely?
qiine
Asking to animate it add an interesting layer of difficulty
svcrunch
I'd like to mention the Little Dorrit Benchmark [1] which I have been running for a couple of years now. It has a few nice features: 1. It tests visual reasoning and structured output in a single task. 2. It seems to sort correctly on advancing general intelligence. As a counterexample, if I'm not misremembering, artificialanalysis.ai made some changes to their benchmark recently after Astra ranked below several older models. 3. While models have gotten significantly better in the past 2 years, the top model is still at 0.78 F1, so the test is not yet saturated. As a reference point, when I started, the top models were in the [0.1, 0.2] range. [1] https://dorrit.pairsys.ai/
eddytrex_
Does a test of instructions how to fold origami figures in a SVG/jpeg exist? Or could be useful?
neilellis
Well that benchmark is now saturated, what next. How fast you can hack the pentagon?
BrokenCogs
Gemini 3.8 flash seems to (subjectively) be the outlier in terms of performance to cost ratio?
villish
The 3 US models have their own style. Qwen3.8 is very clearly distilled from Claude models.
dustfinger
It is interesting how similar the designs are across the models.
dcreater
Why is this a good test?
zaphar
I notice none of the octopi seem to be actually facing the organ.
mock-possum
Try asking an LLM to draw you the cool S.
kennywinker
Would love to see Qwen3.8-27b here, since that is the model most people are running locally.
outlore
Anyone else surprised the generations look so remarkably similar? All of these models have “independently” generalized that the moose should roughly be standing at the same position (left) or that the giraffe should have a certain color palette.
andy_ppp
Love the output from Qwen 3.8 it seems very impressive for the cost! Why does Gemini 3.8 flash blur everything? What are Google playing at!
simonw
I love these.
pohl
Gemini 2.5 Pro is the only model with a sense of where a ferris wheel operator would be.