He is basically an AI professor for law. This study just confirms his existence: https://juliannyarko.com/ Stanford and its donors of course want to replace anyone but its administrators, so they cheer on such anti-intellectual nonsense.
king_zee
I think there will be a market for firms that aggressively market themselves as non-AI, and then as more people turn towards that human connection we'll go full circle
wilg
> In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. 75% win rate seems pretty good! Paper link: https://law.stanford.edu/wp-content/uploads/2026/06/salinas_...
causal
As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.
Esophagus4
Yeah this could be interesting. A lot of the spotlight has been on “law firm stuff” like demand letters and writing contracts… But imagine if a dev team didn’t have to go engineer -> product manager -> legal team to get a question answered on local data retention requirements. You could ship that much faster.
homeonthemtn
Personally I think this is very good. One of the hardest things out there is maintaining a society in the face of changing times and it's because law is dense and slow. I think, in the right hands, this could be huge.
bko
Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say it's dumb. It'll be like the Turing test, which we just blew past years ago and no one cared. After all the hand-wringing about sentience and rights of the AI if it passes the Turing test, and now we just have AI bots running 24/7 writing slop. How does everyone else feel?
chewbacha
My best guess is that Gemini was trained on the textbooks that the questions are meant to test against, thus they are probably better at explicit recall of those questions or related questions. This is a pretty limited introductory course based on what it says in the methods of the paper itself.
throw7
Oh, a "Human-Cented" study by AI lover: Julian Nyarko Professor of Law Co-Chair Stanford Law AI Initiative Senior Fellow, Stanford Institute for Human-Cented AI (HAI) LOL!
Thaxll
AI will never convince a jury though.
gaiagraphia
Incredible that the common people will be able to wrestle the right to rule of law away from the bloated legal caste, who have built themselves quite the moat. The inaccessibility of justice is a huge driver of inequality. Any tools which bridge this gap will help make a more just society.
airstrike
Yes, LLMs are great at search. That's not news.
t0lo
Library outperforms student... more news at 9
t0lo
More great news from the prestigious university where 40% of students claim they are disabled https://fortune.com/article/rise-in-elite-students-seeking-a... and where they wanted to ban words such as "chief", "stupid", "karen" and "American" https://reason.com/2022/12/21/stanford-elimination-harmful-l...
Aperocky
> rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. That's the problem, you never know when the 25% deliver a true stink bomb, and that's not considering prompting - while a fair prompt/question maybe considered objective, it's very easy to stray.
KnuthIsGod
In the hands of a domain expert, AI is useful. In the hands of the naive, it is a foot gun. I killed my Arch installation and was stuck at the GRUB prompt.Unwilling to brush up my rusty knowledge of GRUB syntax, I asked Gemini for help. The commands Gemini suggested would have wiped my hd... Once Gemini was told that I was using BTRFS, the suggestion from Gemini looked a bit more sane, but still looked incorrect to me. It was only after I informed Gemini that I was using a NMVE with BTRFS that it finally produced a sane command.
eichi_uehara
I beat lawyers twice before generative AI even existed. Recently I asked Gemini a few questions about personal conflicts in everyday life. It's often too conservative, with views too shallow for the problem. So I still handle human conflicts myself. I only outsource the templated stuff like routine chat replies or marketing copy though it saves me huge amount of time. People who quote AI in serious conflicts are too weak to handle them on their own.
applicative
What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated statement.
quantisan
I'm surprised Stanford Law would go along with this over-reaching press release title. How about "For common first-year contracts-law questions, law professors preferred AI-generated answers to professor-generated answers"
Related Discussions
Found 5 related stories in 109.5ms across 9,294 title embeddings via pgvector HNSW
Discussion Highlights (19 comments)
34981t
He is basically an AI professor for law. This study just confirms his existence: https://juliannyarko.com/ Stanford and its donors of course want to replace anyone but its administrators, so they cheer on such anti-intellectual nonsense.
king_zee
I think there will be a market for firms that aggressively market themselves as non-AI, and then as more people turn towards that human connection we'll go full circle
wilg
> In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. 75% win rate seems pretty good! Paper link: https://law.stanford.edu/wp-content/uploads/2026/06/salinas_...
causal
As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.
Esophagus4
Yeah this could be interesting. A lot of the spotlight has been on “law firm stuff” like demand letters and writing contracts… But imagine if a dev team didn’t have to go engineer -> product manager -> legal team to get a question answered on local data retention requirements. You could ship that much faster.
homeonthemtn
Personally I think this is very good. One of the hardest things out there is maintaining a society in the face of changing times and it's because law is dense and slow. I think, in the right hands, this could be huge.
bko
Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say it's dumb. It'll be like the Turing test, which we just blew past years ago and no one cared. After all the hand-wringing about sentience and rights of the AI if it passes the Turing test, and now we just have AI bots running 24/7 writing slop. How does everyone else feel?
chewbacha
My best guess is that Gemini was trained on the textbooks that the questions are meant to test against, thus they are probably better at explicit recall of those questions or related questions. This is a pretty limited introductory course based on what it says in the methods of the paper itself.
throw7
Oh, a "Human-Cented" study by AI lover: Julian Nyarko Professor of Law Co-Chair Stanford Law AI Initiative Senior Fellow, Stanford Institute for Human-Cented AI (HAI) LOL!
Thaxll
AI will never convince a jury though.
gaiagraphia
Incredible that the common people will be able to wrestle the right to rule of law away from the bloated legal caste, who have built themselves quite the moat. The inaccessibility of justice is a huge driver of inequality. Any tools which bridge this gap will help make a more just society.
airstrike
Yes, LLMs are great at search. That's not news.
t0lo
Library outperforms student... more news at 9
t0lo
More great news from the prestigious university where 40% of students claim they are disabled https://fortune.com/article/rise-in-elite-students-seeking-a... and where they wanted to ban words such as "chief", "stupid", "karen" and "American" https://reason.com/2022/12/21/stanford-elimination-harmful-l...
Aperocky
> rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. That's the problem, you never know when the 25% deliver a true stink bomb, and that's not considering prompting - while a fair prompt/question maybe considered objective, it's very easy to stray.
KnuthIsGod
In the hands of a domain expert, AI is useful. In the hands of the naive, it is a foot gun. I killed my Arch installation and was stuck at the GRUB prompt.Unwilling to brush up my rusty knowledge of GRUB syntax, I asked Gemini for help. The commands Gemini suggested would have wiped my hd... Once Gemini was told that I was using BTRFS, the suggestion from Gemini looked a bit more sane, but still looked incorrect to me. It was only after I informed Gemini that I was using a NMVE with BTRFS that it finally produced a sane command.
eichi_uehara
I beat lawyers twice before generative AI even existed. Recently I asked Gemini a few questions about personal conflicts in everyday life. It's often too conservative, with views too shallow for the problem. So I still handle human conflicts myself. I only outsource the templated stuff like routine chat replies or marketing copy though it saves me huge amount of time. People who quote AI in serious conflicts are too weak to handle them on their own.
applicative
What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated statement.
quantisan
I'm surprised Stanford Law would go along with this over-reaching press release title. How about "For common first-year contracts-law questions, law professors preferred AI-generated answers to professor-generated answers"