Launch HN: Laminar (YC S24) – RL-trained model that finds errors in agent traces
Hi HN, we’re Robert and Din from Laminar ( https://laminar.sh ). We’ve been building open-source observability for AI agents since 2024. Recently, we released flow-1, a model trained with RL to find errors in agent traces. On our benchmark ( https://laminar.sh/blog/flow-1 ), it matches GPT-6-sol in detection quality while being 23x cheaper. This makes it affordable to analyze every production agent trace, without sampling. Here’s a one-minute walkthrough: https://youtu.be/KJa3h6ZJNLQ We also put together a post with example traces and analyses from flow-1 and other models: https://laminar.sh/blog/flow-1-in-action Agent observability is crowded, so instead of a bullet list of how Laminar differs from X, we’ll explain how we’re solving agent observability from first principles. Agents now run for many steps and fail implicitly. An agent can click the wrong button or cite the wrong document while every call completes successfully. These errors are the most useful ones to catch and understand. To find them, you need to understand what the agent was trying to do and inspect its LLM calls and tool results. Simply looking at the final agent output is not enough. Collecting traces alone doesn’t solve this. A lot of agent observability platforms are still data collectors at their core. You can end up paying to store traces that nobody has time to read or extract value from. Using another LLM to inspect traces is THE solution, but capable models are expensive at scale and cheaper ones often miss errors or flag unrelated things. So we built our model to solve exactly this problem. We finetuned a small OSS model and then post trained it with RL on thousands of synthetic investigations of realistic agent workflows in software, legal, customer support, etc. We got a model that is both good at agent trace analysis and very cheap to run. flow-1 runs inside a harness we call Signals. It acts as a coding agent, treating an agent trace like a repo, with each span (an LLM call, tool call, etc.) as a file. It starts from a compressed string representation of a full agent trace and a user prompt, then greps across spans and reads the relevant ones in full. The beauty of Signals is that it can respond to a general prompt. You can ask it to identify user frustration, check whether your browser agent completed a checkout, and so on. It’s not limited to raw error detection. It returns findings as JSON in a schema users can freely define. We call these signal events. Once you’re analyzing every trace, you also have too many individual events to read. That’s why we automatically group similar signal events into clusters. You can see recurring problems and how often they happen, then drill into individual events and the traces behind them. You can use traces from a cluster to build an eval dataset for that type of failure. You can then use your favourite coding agent to hillclimb on those evals and check that the fix works across these cases before shipping it to prod. Trace → analyze → catch errors and form clusters → fix → release. Making this loop easy is why we built Laminar. We’ve put the same thought into the rest of Laminar. Our blog ( https://laminar.sh/blog ) explains how looking closely at how agents actually work led us to build 20x trace compression, a custom SQL query engine and automatic task extraction. Laminar is Apache 2.0 and self-hostable ( https://github.com/lmnr-ai/lmnr ). We make money from the managed cloud. We’d love to hear how you track your agents’ performance today. How much do you learn from user feedback versus your own monitoring? And when you fix something, how do you check that it hasn’t introduced a different failure?
Discussion Highlights (10 comments)
ghm2180
Very cool. I liked the high level overview where you can view it like do a funnel analysis for failures. Good for fast decision making. I wonder if you have thought of using this for the voice AI space for creating traces from many test calls to speed up testing of voice bots? It's true there are a lot of challenges _before_ one gets to the stage of running actual calls like creating good test data etc, but I wonder if you see a definite role for this framework in that space?
SantiDev
Congrats on the product! I love the design of the website and the app. Curious whether it was made in-house, by an external design agency, with AI models, or something else
danielmarkbruce
To answer your question - we use cheaper models to just flag if something looks obviously flawed. That covers the vast majority of things because it's the propagation of obviously stupid stuff that causes a problem for our system. To the extent we are doing something sophisticated, we use an appropriate architecture, like it's built into the system.
BoorishBears
You should try the bullet points then earning someone reading the rest, that's where I bounced.
dmos62
How do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details. Edit: I initially asked also about general tips for observing agents, but later, noticing that there were no interactions with my comment yet, edited my comment to remove that question, because I felt that it was too broad and would detract from my main question. To my surprise, both were answered by op after my edit; kudos!
paulariquelmee
The point about paying to store traces nobody reads hit home. We had the same problem with customer interviews. We did 8 of them and the notes just sat in a doc until we tagged each problem by how many customers it affected. Curious how you decide which flagged errors matter most. Is it how often they show up across traces, or do you weight them by user impact?
nucleus777
Interesting. Most of the failures I've seen aren't the agent doing something obviously daft, it's a tool call that returns fine but the result shouldn't be trusted. Stale cached data, or a web page or file with text in it that's basically trying to give the agent instructions. Everything after that looks reasonable, it's just built on a bad input. Can flow-1 flag that kind of thing? And how does it tell the difference between the tool giving back something wrong and the agent misreading a result that was actually fine? From the trace alone those look pretty similar to me.
scottfalconer
We ran into an interesting side effect of agent performance that we needed to explain in our results to our executive team. In our case we were rolling out an AI support agent that could help resolve customer problems. At the start we had calculated the total time and human cost per support ticket going back a few years. After a few months of running the AI agent, we found that both our time and cost per ticket had increased - but then digging in, the issue was that our overall cost and time had decreased and it was just that the more complicated issues the agent wasn't able to answer were the ones now going to the humans, skewing our per ticket cost up. So in this case it's not that it was a different failure, it was that a signal that seemed to indicate failure was actually a signal of success.
jing09928
这类 trace 诊断如果能直接沉淀成回归测试，落地价值会很高。你们目前更看重单次高风险错误，还是跨 trace 反复出现的模式？
sebastienburel
To your question about how we track ours: one failure I had left no trace at all. A morning run on a resident agent just didn’t happen. The host timer runs on the uptime clock, the Mac slept through the slot, and nothing retries. As far as I can tell from the post, Signals reads traces that exist. Do you do anything for the run that should have happened and didn’t, like diffing against an expected schedule?