TaxProf Blog

Jonathan Choi (Minnesota; Academic google), Kristin Hickman (Minnesota; Academic google), Amy Monahan (Minnesota) and Daniel Schwarcz (Minnesota; Academic google), ChatGPT goes to law school:

How well can AI models write law school exams without human help? To find out, we used the widely publicized ChatGPT AI model to generate answers on four real exams at the University of Minnesota Law School. We then score these exams blindly as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving low but passing grades in all four courses. After detailing these results, we discuss their implications for legal education and advocacy. We also provide example prompts and tips on how ChatGPT can help with legal writing.

Overall, ChatGPT passed all four classes based on his final exam, with a C+ average on all exams, a result that would grant credit for the JD but would place the student on academic probation. In particular, if such performance were consistent across law school, the scores earned by ChatGPT would be enough for a student to graduate. Despite performing well enough to theoretically earn a JD degree, ChatGPT generally scored at or near the end of each class. ChatGPT received a B in Constitutional Law (36th of 40 students), a B- in Employee Benefits (18th of 19 students), a C- in Taxation (66th of 67 students), and a C- in Liability ( 75th of 75 students). …

The following figures show ChatGPT’s performance on each question (or, in the case of multiple choice questions, each set of questions) relative to actual students. The figures are density plots, where the x-axis reflects the score for each test component and the y-axis reflects the proportion of students who received the corresponding score. The black dashed lines show the average scores of all students and the red solid lines are the ChatGPT scores. The ChatGPT percentile performance for each question is also shown in red.