CNN
—
ChatGPT is smart enough to pass the prestigious graduate-level exam—although the score is not particularly high.
The powerful new AI chatbot tool recently passed a law exam in four courses at the University of Minnesota and another exam at the Wharton School of the University of Pennsylvania, according to school professors.
To test ChatGPT’s ability to generate answers to exams in these four courses, professors at the University of Minnesota School of Law recently scored the exams blindly. After completing 95 multiple-choice questions and 12 essay questions, the robot achieved a grade point average of a C+ student, with lower but passing grades in all four courses.
ChatGPT performed better on Wharton’s business administration course exam, earning grades of B to B-. In a paper detailing the performance, Wharton professor Christian Terwiesch said that ChatGPT does a “remarkable job” at answering basic operations management and process analysis questions, but fails at handling more advanced underperformed when prompted and made “surprising mistakes” in basic math.
“The scale of these errors can be substantial,” he wrote.
The test results come as more schools and teachers express concern about ChatGPT’s immediate impact on students and their ability to cheat. Some educators are now rethinking their assignments with alarming speed in response to ChatGPT, even though it’s unclear how widely the tool is used among students and how much it actually harms learning.
Member of Congress delivers speech written by AI
Since its launch in late November, ChatGPT has been used to generate original articles, stories and lyrics based on user prompts. The abstracts of research papers it drafted fooled some scientists. Some CEOs even use it to write emails or do accounting.
ChatGPT is trained on a fixed amount of online data to generate responses to user prompts. While it has gained traction among users, it has also raised some concerns, including about inaccuracies and their potential to perpetuate bias and spread misinformation.
Jon Choi, one of the University of Minnesota law professors, told CNN that the purpose of the test is to explore the potential of ChatGPT in assisting lawyers in their practice and helping students take exams, whether or not their professors allow it, because the questions are often modeled after lawyers in the exam. Writing done in real life.
“ChatGPT grapples with the most classic parts of law school exams, such as spotting potential legal issues and in-depth analysis that applies legal rules to the facts of a case,” Choi said. “But ChatGPT can be very helpful in making a first draft that students can then refine.”
He believes that human-machine collaboration is the most promising use case for ChatGPT and similar technologies.
“My strong hunch is that AI assistants will become a standard tool for lawyers in the near future, and law schools should prepare students for that possibility,” he said. “Of course, if law professors want to continue testing simple recall of legal rules and teaching, they will need to enforce restrictions, such as banning the internet during exams, to enforce that.”
Similarly, Wharton’s Terwiesch found that chatbots are “remarkably good” at modifying answers based on human cues, such as after pointing out mistakes, suggesting that it’s possible for people to collaborate with AI.
Scott Galloway on the ‘scary part’ of AI tools like ChatGPT
In the short term, however, the question of whether and how students should use ChatGPT remains. For example, public schools in New York City and Seattle have banned students and teachers from using ChatGPT on the district’s network and devices.
Given ChatGPT’s above-average performance on the test, Terwiesch told CNN that he agrees that restrictions should be placed on students when they take the test.
“The ban is necessary,” he said. “After all, when you degree doctors, you want them to know medicine, not how to use a robot. The same applies to other skill certifications, including law and business.”
But Terwiesch believes the technology will eventually still have a place in the classroom. “If all we end up with is the same education system as before, then we’re wasting a great opportunity with ChatGPT,” he said.