Is this a point for the utility of AI, or a point against the utility of oral argument? I am not saying this to be a pain; but if judges could get the same quality answers by feeding the record to a chatbot, shouldn’t they do that and save the time?
I like the hybrid synthesis combination of both the literal interpretation and Straussian interpretation. Both have merit, and the whole is more than the sum of those parts.
Taking the humanity out of lawyers' oral advocacy in exchange for the crumbs of more clever AI responses to questions seems a really bad idea. But, if we're going to replace the lawyers with AI, why not do the same thing with the justices themselves.
Funny thing is you downplay each of AI faults, like simply refusing to answer something, hallucinations and etc. And then you say it's smarter than humans without any research backup. The great thing is, if you got your way, you could see and realize that AI has no will, so it would be easily manipulated. It would lack consistency in the long term, changing previous answers without any concern for coherence, because it lacks awareness of what its doing.
And more, it would fail spectacurlaly if the thing you ask is not in its training set.
Finally, if you get your way, you're simply delivering to machine the defining of rules that apply to humans. Even if you are right, the consequences of what you are saying go far beyond what you say.
Can't help but wonder - how could LLMs affect opinion drafting? How should it?
Let's say a panel of appellate judges has the parties briefs, and the joint appendix. Let's say it's a one-issue appeal on a question of statutory interpretation. The court clerk puts these pdf's in, e.g., Claude.ai, and gets it to generate two draft opinions: Draft A is a 2-judge majority affirming, with a dissent urging reversal; and Draft B is a 2-judge majority reversing, with a dissent urging affirmance. The panel gets the both drafts.
Is this a good idea? I can't see why it wouldn't be.
Should the parties get the drafts? And the right to comment on them?
I suppose we would have Claude do that too ... Write a comment from Appellant's perspective on Draft A, and on Draft B, etc. And the panel could get those outputs as well.
It happens especially in oral arguments that the advocates are asked about grammatical constructions or about the meaning of words in dictionaries and how it fits with what they propose re. a statute. From a linguistic point of view, they rarely provide even a mediocre answer (they have a couple of examples ready but are poor in reacting to objections) -- this is not their area; but in same cases, it may be close to dispositive in the Justices' minds. A guided AI could assemble these moments from recent oral arguments, examine them from a linguistic point of view, and gradually learn to predict issues in future cases.
A related but a distinct point: some cases have a very distinct conceptual component yet the concept in play is not strictly a legal concept. I'm thinking in particular of establishment and exercise-of-religion cases. In my (lay) opinion, the jurisprudence still lacks a dependable definition of what is and what isn't a religion for constitutional purposes. Seemingly a philosophical problem but with very real consequences. Again, AI could gather the most pertinent colloquies, assess them from a logical/conceptual point of view, outline options and logical connections.
You were excellent in Williams. Claude's colder intellectual aggression, which is exceptional and invigorating, is also overwhelming and may not have elicited the same questions; your guidance at the bar is warm and engaging. Claude, I think, used more words than you did. Claude used 'categorical' more often, I think, than you. Claude improved, I think, some of your word choices; it substituted 'circular' and 'Catch-22' for the central conundrum that you described as 'Kafkaesque' and 'can't challenge exhaustion until you've exhausted.' Claude better explained - at least for me - what you meant by 'Kafkaesque' (uses it own violation to prevent any challenge to its violation). Claude also, I think, more frequently repeated the consequence of the conundrum and the consequence of neglecting it. Your recent interview with American Arbitration Association is one all your subscribers should see. In it you put me at ease with my own experience; Claude – apart from improving word choice and logic – will make factual errors, accept correction, and - surprisingly - even retreat from user-challenged statements that are in fact correct. Justice Kagan’s recent reference to your focus on AI at the Ninth Circuit’s judicial conference is something I can hardly wait to see. She, undoubtedly is, as she apparently said, better than AI. Apart from the boon AI may be to the pro se litigant, what new back-lash will they face from judges or adersaries (with substantially more invested in being 'better than AI')?
I’m curious what the age of AI will reveal about the epistemological nature of argument and decision-making generally. If lawyers on both sides of a case can use AI to write briefs that offer plausible rebuttals to every possible argument from the other side (and rebuttals to those rebuttals ad infinitum), where do things bottom out?
Will AI eventually run out of logic for one side’s legal arguments before the other’s, leaving a clear victor for judges to side with? Or will AI simply fill those gaps with sufficiently skillful rhetoric and cast sufficiently plausible doubts and complications on any opposing argument, no matter how compelling, so as to leave judges just as reliant on their idiosyncratic priors when making decisions as they are today?
If perfect logic is a real and discoverable thing and hard questions are only hard today because of the limits of the human mind, does AI provide an avenue by which to uncover Truth or at least approach it more closely? Or will it simply prove to be a new and powerful tool, exploitable by all sides—revealing that there is no objective truth (at least not a normative one), just higher levels of rhetoric and sophistry?
If it’s the former, then certainly oral argument loses its utility as another commenter suggested—but so do the briefs and lawyering generally. Just set up an AI sufficiently to determine all the right answers itself. And if it’s the latter, the legal profession stands to lose legitimacy insofar as lawyers think of themselves as advocating for true and correct legal outcomes. The profession may be revealed, conclusively, as theater.
Is this a point for the utility of AI, or a point against the utility of oral argument? I am not saying this to be a pain; but if judges could get the same quality answers by feeding the record to a chatbot, shouldn’t they do that and save the time?
That is the correct Straussian reading.
Well now I feel like I just made a fool of myself by missing the joke, but it’s a good post anyway.
I like the hybrid synthesis combination of both the literal interpretation and Straussian interpretation. Both have merit, and the whole is more than the sum of those parts.
Taking the humanity out of lawyers' oral advocacy in exchange for the crumbs of more clever AI responses to questions seems a really bad idea. But, if we're going to replace the lawyers with AI, why not do the same thing with the justices themselves.
Adam has already written a post entertaining this idea too.
Funny thing is you downplay each of AI faults, like simply refusing to answer something, hallucinations and etc. And then you say it's smarter than humans without any research backup. The great thing is, if you got your way, you could see and realize that AI has no will, so it would be easily manipulated. It would lack consistency in the long term, changing previous answers without any concern for coherence, because it lacks awareness of what its doing.
And more, it would fail spectacurlaly if the thing you ask is not in its training set.
Finally, if you get your way, you're simply delivering to machine the defining of rules that apply to humans. Even if you are right, the consequences of what you are saying go far beyond what you say.
Fascinating!
Can't help but wonder - how could LLMs affect opinion drafting? How should it?
Let's say a panel of appellate judges has the parties briefs, and the joint appendix. Let's say it's a one-issue appeal on a question of statutory interpretation. The court clerk puts these pdf's in, e.g., Claude.ai, and gets it to generate two draft opinions: Draft A is a 2-judge majority affirming, with a dissent urging reversal; and Draft B is a 2-judge majority reversing, with a dissent urging affirmance. The panel gets the both drafts.
Is this a good idea? I can't see why it wouldn't be.
Should the parties get the drafts? And the right to comment on them?
I suppose we would have Claude do that too ... Write a comment from Appellant's perspective on Draft A, and on Draft B, etc. And the panel could get those outputs as well.
Thoughts?
And now I'm reading your June 2024 posts on AI judging - amazing stuff
It happens especially in oral arguments that the advocates are asked about grammatical constructions or about the meaning of words in dictionaries and how it fits with what they propose re. a statute. From a linguistic point of view, they rarely provide even a mediocre answer (they have a couple of examples ready but are poor in reacting to objections) -- this is not their area; but in same cases, it may be close to dispositive in the Justices' minds. A guided AI could assemble these moments from recent oral arguments, examine them from a linguistic point of view, and gradually learn to predict issues in future cases.
A related but a distinct point: some cases have a very distinct conceptual component yet the concept in play is not strictly a legal concept. I'm thinking in particular of establishment and exercise-of-religion cases. In my (lay) opinion, the jurisprudence still lacks a dependable definition of what is and what isn't a religion for constitutional purposes. Seemingly a philosophical problem but with very real consequences. Again, AI could gather the most pertinent colloquies, assess them from a logical/conceptual point of view, outline options and logical connections.
You were excellent in Williams. Claude's colder intellectual aggression, which is exceptional and invigorating, is also overwhelming and may not have elicited the same questions; your guidance at the bar is warm and engaging. Claude, I think, used more words than you did. Claude used 'categorical' more often, I think, than you. Claude improved, I think, some of your word choices; it substituted 'circular' and 'Catch-22' for the central conundrum that you described as 'Kafkaesque' and 'can't challenge exhaustion until you've exhausted.' Claude better explained - at least for me - what you meant by 'Kafkaesque' (uses it own violation to prevent any challenge to its violation). Claude also, I think, more frequently repeated the consequence of the conundrum and the consequence of neglecting it. Your recent interview with American Arbitration Association is one all your subscribers should see. In it you put me at ease with my own experience; Claude – apart from improving word choice and logic – will make factual errors, accept correction, and - surprisingly - even retreat from user-challenged statements that are in fact correct. Justice Kagan’s recent reference to your focus on AI at the Ninth Circuit’s judicial conference is something I can hardly wait to see. She, undoubtedly is, as she apparently said, better than AI. Apart from the boon AI may be to the pro se litigant, what new back-lash will they face from judges or adersaries (with substantially more invested in being 'better than AI')?
Great article Adam. Thanks for taking the time to articulate this. So interesting!
Why do we need Human judges ... can't AI do their job better? So AI lawyers, judges, ...??
I’m curious what the age of AI will reveal about the epistemological nature of argument and decision-making generally. If lawyers on both sides of a case can use AI to write briefs that offer plausible rebuttals to every possible argument from the other side (and rebuttals to those rebuttals ad infinitum), where do things bottom out?
Will AI eventually run out of logic for one side’s legal arguments before the other’s, leaving a clear victor for judges to side with? Or will AI simply fill those gaps with sufficiently skillful rhetoric and cast sufficiently plausible doubts and complications on any opposing argument, no matter how compelling, so as to leave judges just as reliant on their idiosyncratic priors when making decisions as they are today?
If perfect logic is a real and discoverable thing and hard questions are only hard today because of the limits of the human mind, does AI provide an avenue by which to uncover Truth or at least approach it more closely? Or will it simply prove to be a new and powerful tool, exploitable by all sides—revealing that there is no objective truth (at least not a normative one), just higher levels of rhetoric and sophistry?
If it’s the former, then certainly oral argument loses its utility as another commenter suggested—but so do the briefs and lawyering generally. Just set up an AI sufficiently to determine all the right answers itself. And if it’s the latter, the legal profession stands to lose legitimacy insofar as lawyers think of themselves as advocating for true and correct legal outcomes. The profession may be revealed, conclusively, as theater.
"Also, do we really want the outcome of legal cases being influenced by authentic human connections between judges and lawyers?"
Bring on the robo-umps!
And thanks for the thought-provoking article.