When LLM judges agree, should we believe them?

Tsarp

Kinda weird to generalize "LLM". Every lab, every model is different. Has its own biases, reward functions etc.

troupo

Without reading the article (doesn't matter if it's pro or contra): no, of course not. It shouldn't even be a debatable question.

bryzaguy

They would all agree raspberry has two Rs

Founderarcstone

Great point this will be interesting how this develops.

qarl

While this is absolutely true - I'd hesitate to discount using similar agents for checking each other. Two agents will almost never hallucinate in the same way, regardless of their weights - and by having a second one (with a different context) check almost entirely eliminates the problem.

VaradD09

I believe it depends on the LLM itself. Like what model as each model has diff weights and diff data trained onn

ex1fm3ta

I kinda find it funny when I use the advisor on claude code and it agrees with the ideas that the previous model did. For info: the advisor(s) available are higher end models. For example: you use sonnet, the available advisors are opus and fable. If you use Haiku, the advisor are sonnet, opus and fable.

mey

A shared blind spot as noted at the bottom needs to be considered more often. In my day job, most of my coordination with others and now LLMs, is clarifying context and requirements. Claude is very happy to make assertions without the full picture in my experience, even when I give it as much context as I can.

nekusar

Betteridges law of headlines says "NO"

peter_d_sherman

Observation: A panel of Judges (multiple Judges), whether it's multiple AI's or not, is fundamentally -- a Jury !

lhk931122

Since the baseline LLM architecture is similar with each other, maybe already there are relations between their each opinion. Of course, this is an assumption and the explicit training seems effective in this case. But, I'm curious whether the approach would be effective in other cases (in terms of generalization?)