I love learning new things, Khan academy got me all the way through college, and I use ChatGPT / Claude to help me study papers regularly. I got frustrated really fast. Here is an example:

https://learnyourway.withgoogle.com/scopes/1KNlGW5E/immersiv...

It starts with just this sentence, followed by a quiz on that sentence:

> When we are born, we inherit our genetic makeup and biological features. However, our identity as human beings develops through interactions with others in society. Many experts in both psychology and sociology have described the process of self-development as a key step to understanding how that "self" learns to function within society.

Followed by the quiz:

> Question 1: Based on the provided text, what is a key difference in focus between psychology and sociology regarding self-development? A) Sociology is concerned with inherited traits, whereas psychology is concerned with societal norms. B) Psychology studies societal functions, while sociology studies individual identity. C) Psychology focuses on genetic makeup, while sociology focuses on social interactions. D) Both fields exclusively study the biological features inherited at birth.

I thought D makes the most sense, as nothing in the immediate text provides a more granular answers. But it's not D. It made me question my intelligence, maybe I misread the sentence, maybe I needed to read something else? Oh there is a button for the entire PDF, but then isn't the purpose of it to break down the PDF into chunks and ask me questions on what I'm reading?

I'm sure this is a fixable bug, but I was looking for the "provide feedback" button, there is none.

This would be very frustrating to a student.

It isn't you. None of those answers are correct. Sociology studies societies and cultures; collective behaviors at different scales within different niches, etc. It's an LLM hallucinating again.

Ignoring that, NONE of those answers are correct. It wants "C) Psychology focuses on genetic makeup, while sociology focuses on social interactions.", but that's not a true statement. Psychology is absolutely not focused on genetic makeup.

There is no way to answer that question, period. Neither the term "psychology" nor the term "sociology" is defined in the text (and the terms are used in union, not constrasted), so to have any hope of answering at all you need to apply knowledge not found in the text, which is expressly prohibited.

(edit to add:)

This is just regular poor-quality language model output. The language model is trained on data where the phrase "based on the provided text" makes a common appearance between a text segment and subsequent questions, but the model has no knowledge of the very specific, limiting meaning of that phrase: it limits the following question to assess reading comprehension only, not general knowledge.

So no, I don't think this is a minor bug that's easily fixable: a pure language model will always associate key signaling phrases with the wrong type questions, because it has no concept of (didactic) mode. It basically considers all phrases ornamental instead of purposeful.

> I'm sure this is a fixable bug

I'm sure it is, but the bar for accuracy in education is way too high for a mistake as blatant as this to be allowed to slip through.

(I'm sure someone will chime in about their useless Hum 10 teacher making an even bigger gaffe, and how this can be excused, it's just a beta...)