Show HN: Strot – Compile an LLM's intent once, run decisions locally

solohunter001

Hi HN. STROT compiles a decision task — "route these tickets", "score this feedback", "classify this text" — into a standalone pipeline. A large LLM is used once, at compile time, to synthesize the labels, the input projection, and the handlers. At runtime a small local model (144M, CPU) makes every decision: ~60ms/record, $0 marginal, zero LLM calls, bounded typed-JSON output. The goal was to move the expensive, nondeterministic part (an LLM) entirely to build time and keep runtime frozen, cheap, local, and auditable — the way a compiler moves work off the hot path. What I think is actually interesting, beyond "small model does inference": - The compiler measures instead of guessing. It scores candidate labelings on a teacher-labeled sample and keeps the winner; on the identical model that's +18pp on a routing benchmark (92.5% vs 74.8%) and +14pp on text classification (86.7% vs 72.7%) over naive integration. - It fits a conformal-calibration threshold at compile time, so low-confidence records abstain with a marginal error guarantee rather than guessing. - A golden-record gate rejects any compile that fails known-answer examples — synthesis is stochastic, and I show that variance in the benchmarks. What it doesn't do: it can't make a small model understand things it fundamentally can't. On tweet hate-speech it hits the model's ceiling; the fix there is a task-trained model in the swappable slot, not the compiler. All of it — including the losses — is in the repo's benchmarks with reproducible scripts. There's a browser demo (a 144M model ( which is changeable ) playing Pac-Man, one decision per frame) and a quickstart that runs in a few lines. MIT licensed, pure-stdlib core. Happy to go deep on the calibration math, the compile/cache design, or where this does and doesn't make sense.

jMyles

I think that "infer once, repeat resulting logic forever" is one of the most important / sustainable / humane ways to imagine how AI acclimates to the universe. https://pickipedia.xyz/wiki/PickiPedia:Artificial_Dumb

wonnage

> If a ghost is close (nearest_ghost_distance small, " "within 2 steps), choose flee_ghost. Otherwise choose " "chase_dot to eat the nearest dot." Dude, this is AI psychosis. Just ask the LLM to write you a program (“compile”) and then you can run it locally forever

publlus_enigma

I love the concept, but tried reading the README on the got, but I am still unclear about precisely what it does or how it does it. Some more detail would be welcome.