ChatGPT gets a passing grade on 4 Minnesota Law exams

ChatGPT, which was unveiled in November as the latest creation by OpenAI, is taking the world by storm — legal education included.

Since its launch, academics have begun experimenting with how ChatGPT may perform on questions from law exams and essays.

This last semester a group of law professors from the University of Minnesota tested the chatbot on four law exams. The professors blindly graded the program’s responses along with other test responses from their students, according to their research paper.

Though it didn’t beat most of the scores of actual students, the chatbot received a passing grade. Overall they found that ChatGPT averaged a C+ across all the exams compared to the actual students’ B+ average.

ChatGPT received a B in Constitutional Law (36th out of 40 students), a B- in Employee Benefits (18th out of 19 students), a C- in Taxation (66th out of 67 students) and a C- in Torts (75th out of 75 students), according to the research paper.

“ChatGPT was fairly reliable at describing legal rules, which was a surprise; lots of folks have reported ChatGPT making up facts,” said Jonathan Choi, lead author of the research paper. “Many people might be surprised to hear that ChatGPT is very bad at math. Its worst performance on the exams was on multiple-choice questions involving mathematical reasoning.”

“I think the most interesting finding is that ChatGPT struggles with spotting potential legal issues and with deep legal analysis,” he continued. “I expect these skills will continue to be a core human competency for some time.”

The research shows that the program generally performed better on the essay portion over the multiple choice but the professors still described its essay performance as “highly uneven,” adding that when the essay questions were incorrect, they were “dramatically incorrect.”

Choi shared that a recent informal poll of Stanford students found that 17% of them used ChatGPT in their finals last semester. Just a few weeks after the program was released.

“If law students aren’t already using these tools during exams, they certainly will soon,” Choi said. “Until we can change the way that law school courses are taught, I expect law professors will either ban the use of ChatGPT or ban internet access during exams.”