Susan Tanner, who teaches legal writing at University of Louisville Louis D. Brandeis School of Law in Kentucky, an early adopter of GenAI, advocates for law students and professors to learn about the technology’s capabilities and limitations.
“I’ll tell you, my students are not always enthusiastic users of ChatGPT. I’m surprised at how much convincing I have to do to have them just play around with it a little bit,” she said. GenAI is no substitute for learning, Tanner said. Students still need to know the coursework and case law, but GenAI can be a valuable implement for putting those concepts together in a meaningful way.
“ChatGPT is really good at writing something that looks true. And if students don’t have an understanding of whether or not that is true, it’s harder to check their work,” said Tanner, adding that skepticism is to be expected when a new resource is introduced. “I used to teach English at a time when people were afraid to use Wikipedia because they were told over and over and over, ‘Don’t use it; it’s going to be wrong.’”
The inevitable ubiquity of GenAI in the legal profession means students need to understand it, and law professors need to be willing to teach it. That, too, goes against the grain of traditional legal education.
“You have to understand that professors really want to be comfortable as experts in something. And no one is an expert in using AI for law yet,” Tanner said.
To that end, she is working to create a toolkit to help fellow law professors incorporate GenAI into their legal writing curricula. Tanner said she hopes to generate a framework for explaining GenAI processes and how the technology can be used in the classroom.
For instance, the toolkit could offer suggestions on how to input a series of prompts into GenAI to generate a good hypothetical scenario for law students to consider in classroom discussions or on a test.
Tanner is collaborating with fellow law professor Tracy Norton and William Monroe, the school’s assistant director for instructional technology. The toolkit should be available online for free in Fall 2024.
At Southwestern Law School in Los Angeles, faculty have participated in training sessions and workshops on the benefits and drawbacks of GenAI. Professor Cristina Knolton, who teaches legal analysis and writing, is leading several initiatives to incorporate AI into the school’s curriculum.
“Students have to develop a fundamental basic understanding of legal writing, so we need to make sure that they’re not relying too heavily on ChatGPT for all of their assignments,” Knolton said. “But we also believe that it is going to very likely change the legal landscape. By the time they enter the practice of law, they’re going to need to know how to use generative AI and how to effectively assess it for accuracy, for relevance, for all the things that we’re seeing have trouble right now with AI.”
Knolton takes her law students through a series of exercises to evaluate AI, including giving ChatGPT fictional client facts to see if the technology can accurately summarize a mock legal case.
Students are asked to look for relevance, bias, tone, organization and writing style. She has also shown students how ChatGPT synthesized existing statutes.
“Accuracy was a big problem,” Knolton said. “It would actually misstate the law.”
In another exercise, Knolton allowed students to try GenAI on a class paper. “They turned in a draft, and they had to put at the end of their paper: ‘Yes, I used it. Here are the places that I used it. And I thought it was effective for this reason,’” she said. “Or, if you didn’t use it, you still had to write a paragraph about why you chose not to use it, whether you would use it next time, or how you think you might find it effective in helping you in the future.”
Knolton said she can easily tell the difference between a student’s writing and that created through ChatGPT. Even so, she sees the technology’s merits: GenAI can help students ensure that their writing is clear, grammatically correct and well organized.
“But for a lot of students, I recommended going back to their own writing style, which was much simpler and more straightforward,” Knolton said. “They were over-complicating things because they were relying too heavily on ChatGPT.”
Ready access to AI tools online means law schools will likely modify how they give tests, possibly proctoring exams or limiting internet access.
Professors are also working to make sure tests genuinely assess students’ thinking processes.
“We really want to preserve that thinking and not have ChatGPT and generative AI do all the thinking for you,” Knolton said.
How does AI perform on exams?
Law professors have expressed a lot of concern about students using ChatGPT on exams. Studies show that AI is only as good as the user.
A group of professors at University of Maryland tested GPT-4 during the spring of 2023 and found that it performed better than ChatGPT or GPT-3.5 on tasks such as test-taking.
To test GPT-4’s abilities, the professors ran it on final exams and graded its output alongside students’ performance. They found that GPT-4 produced smoothly written answers but failed to spot many important issues, much like a bright student who had neither attended class often nor thought deeply about the material.
It uniformly performed below average in every course.
Two law professors, Jonathan Choi of University of Southern California, Gould School of Law and Daniel Schwarcz of University of Minnesota Law School, administered law school exams to students with and without access to GPT-4.
They found that assistance from GPT-4 enhanced performance on multiple-choice questions but not on essay questions.
They also found that GPT-4’s effect depended heavily on a student’s starting skill level; students at the bottom of the class saw huge performance gains with AI assistance, while students at the top of the class saw performance declines.
They concluded that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers.
In addition, they graded exams written by GPT-4 to compare them with those written by humans alone and those written by humans with assistance from AI. With basic prompts, GPT-4 was a mediocre student, but with optimal prompting it outperformed both the average student and the average student with access to AI.
Read part one of this premium three-part series: How AI is changing lawyering.
