CC-Canary: Detect early signs of regressions in Claude Code

aleksiy123

Interesting approach, I've been particularly interested in tracking and being able to understand if adding skills or tweaking prompts is making things better or worse. Anyone know of any other similar tools that allow you to track across harnesses, while coding? Running evals as a solo dev is too cost restrictive I think.

evantahler

I feel like asking the thing that you are measuring, and don’t trust, to measure itself might not produce the best measurements.

wongarsu

See also https://marginlab.ai/trackers/claude-code-historical-perform... for a more conventional approach to track regressions This project is somewhat unconventional in its approach, but that might reveal issues that are masked in typical benchmark datasets

Retr0id

What is "drift"? It seems to be one of those words that LLMs love to say but it doesn't really mean anything ("gap" is another one).

redanddead

the actual canary is the need for the canary itself

ctoth

A useful(ish) trick I've found is adding a persona block to my CLAUDE.md. When it stops addressing me as 'meatbag' I know the HK-47 persona instructions are not being followed, which means other instructions are not being followed. Dumb trick? Yup. Does it work? Kinda? Does it make programming a lot more fun and funny? Heck yes. Don't lecture me on basins of attraction--we all know HK is a great programmer.

jdiff

My attitude towards this is growing similar to my attitude towards Windows. If I have to fight against my tools and they are actively working against me, I'd rather save the sanity and time and just find a new tool.

majormajor

In addition to the elsewhere-mentioned "you're using a black box to try to analyze the same black box," the fundamental metrics all seem incredibly prone to other factors than any Claude Code changes. Claude Code changes all the time—it's the whole shitty trend of the day—but you can't tell which of those changes are better or worse from analyzing results on independent novel tasks. And you're baking in certain conclusions: "HOLDING / SUSPECTED REGRESSION / CONFIRMED REGRESSION / INCONCLUSIVE". Where's an option for "better than previous baseline"? Seems certainly possible that a session could have better-than-average numbers on the measured things. Overall, though, there's just so much here that's just uncontrolled. The most obvious thing that isn't controlled for is the work itself. What does the typical software project look like? A continued accumulation of more code performing more features ? What's gonna make an LLM-based agent have to do more work? Having to deal with a larger, more complicated codebase. Nothing in this seems to attempt to deal with the possibility that a session that got labeled a regression might have actually been scored even lower against a month ago's Claude Code. "It's harder to read code than to write code" and "codebases take more effort to modify over time as they grow" are ancient observations. Drift detection would require static targets and frequent re-attempts. I use it everyday and haven't seen worsening. (It's definitely not static but the general trend has been good.) But I use it on a codebase that was already very complex before we started using these tools, where overall every three months or so has brought significant improvements in usability and accuracy.