Jev and System One Models: Calibration Beats Accuracy

darvo31

Spot on. A slightly less accurate but calibrated model inspires far more trust than a 'perfect' one that's consistently overconfident.

edot

Humans don’t use “quietly” in normal parlance. Come on man, try harder. Edit: Dude, you didn’t even use the thing you’re talking about? It’s on OpenRouter. Do better! Edit 2: OP is a ~60 day old account, only other (positive) commenter is a ~48 day old account. Sus.

svg7

Calibration has to be measured on your dataset. Just because the probs sum to 1, does not make Jev or Jev-like models claibrated. For folks interested in digging deeper into calibration, studying ad click prediction models (where calibration is super important) is a good place to start.

rahimnathwani

The way he proposed to deal with "How urgent is this customer support message?" is "a score over ordered levels". I'm not sure that's a good way to use Jev for this use case. If I had this problem, I would ask Jev to answer several different yes/no questions about each message, and then use the probabilities as inputs into a logistic regression model that predicts urgency. If you ask the model specific yes/no questions which can be answered reasonably objectively from the input, I think the answers are going to be more stable over successive generations of models. e.g. if you ask 'Is the customer angry?' I'd expect that answer to have high agreement between models and between models and humans. But directly answering the 'is it urgent' question is much harder. (Although I suppose you can try to put the rules in the prompt.)

ram_rar

> “Zero hallucination.” This is the part that bothers me the most. How is it 0 hallucination, if the correct answer is not even the part of the options. There is no way to mark absentia or a way to know I absolutely cannot choose any of the options. I am wondering, if anyones tried dead simple combinations of embedding with logistic regression to solve classifications problems?

xzasdfghjkl

Let me state it plainly --- this is not just a badly written article. It's super annoying and content free.

amluto

It seems to me that Jev cannot be usefully calibrated out of the box, in a fairly strict sense. Suppose I have a pull request and the state is the title, description, etc. The question is “Will it be merged?” (it doesn’t really matter whether it’s a choice or a “noul” [0]). Now consider that two different projects may have radically different criteria for accepting a PR. And the two projects may have different probabilities for acceptance of a random PR from the distribution of PRs they get (i.e. the overall fraction of PRs that are accepted). So what could Jev possibly return that is “calibrated” for both? It doesn’t even have an “I have no idea” option because the output schema cannot distinguish between “I am confident that there is a 50% probability that the answer is yet conditioned on the state” and “there is no useful information contained in the state that I can extract and therefore you should assume that you posterior distribution is the same as your prior”. For fun, I gave Jev some irrelevant state and asked it various questions for which the state was useless (I picked sporting outcomes), and it was 0-for-3 at giving yes/no probabilities that were particularly close to the obviously correct no-information answers or close to 0.5 in cases where the prior was far from 0.5. This is a silly test, but I’ve personally encountered genuine production situations where the best classifier available (or at least the best one available at any cost remotely close to what it was worth) was, drumroll please, a constant. But it was a calibrated constant: we measured it! And there is no way to feed this sort of information to Jev. (Yes, I tried it. Even literally stating the distribution in the state does not work well, although it does appear to have some effect on the outputs.) [0] Is “noul” even a word? I know what a binary classifier is…