Replying to
@pluralistic@mamot.fr Thank you for a new brilliant post! LLMs can be very useful in teaching, if they're used as "possible crackpots". I teach BSc physics and statistics; a frequent kind of exercise my students get is to analyse an LLM's output to a physics/statistics question or problem, and determine where it's correct and where it's wrong, explaining *why* those parts are correct or wrong. This puts the students with their back against the wall: they understand that using another LLM to do the exercise would just be running in circles.
This kind of exercise is useful exactly for the reasons you discuss in your post. Some LLM's errors are coarse and evident, but others are very subtle. Examining (first together with the teacher) and looking for this kind of errors is a way to learn and test *discernment* of subtler concepts. Some students even get a sense of empowerment in spotting subtle LLM errors in a subject they're learning – and that's the goal indeed!
Variations of this kind of approach can be made ad libitum. Give the same prompt to two different LLMs, or several times to one LLM; notice the differences; determine which (if any) is correct.
Similarly with summaries: prompt an LLM to summarize a given section or chapter from the textbook, then the students must find out what's wrong, or which important pieces have been omitted, in the output summary.
These kinds of exercise also make the students aware that LLMs are not fully reliable, even when what they're spitting out sounds technical and "expert".
All this also connects to a major pre-LLM problem, nicely expressed by Feynman: "I don't know what's the matter with people: they don't learn by understanding; they learn by some other way – by rote, or something. Their knowledge is so fragile!" (similar sentiments were expressed by many other earlier physicists, like Maxwell). Our education system has favoured memorization instead of discernment, for decades or centuries. "Do we, in our schools and colleges, foster the spirit of inquiry, of skepticism, of adventurous thinking, of acquiring experience and reflecting on it? Or do we place a premium on docility, giving major recognition to the ability of the student to return verbatim in examinations that which he has been fed?" (R. E. Gibson). The appearance of LLMs is, finally, making this rote-learning inadequate, and discernment more and more necessary.
#LLM #education
