This is just trivia. I would not use it to test computers -- or humans.

littlestymaar 3 days ago | parent | next [-]

It's good way to assess the model with respect to hallucinations though.

I don't think a model should know the answer, but it must be able to know that it doesn't know if you want to use it reliably.

	▲	esafak 3 days ago \| parent [-]
		No model is good at this yet. I'd expect the flagships to solve the first.

▲

parineum 3 days ago | parent | prev [-]

Everything is just trivia until you have a use for the answer.

OP provided a we link with the answer, aren't these models supposed to be trained on all of that data?

▲

esafak 3 days ago | parent | next [-]

There is nothing useful you can do with this information. You might as well memorize the phone book.

The model has a certain capacity -- quite limited in this case -- so there is an opportunity cost in learning one thing over another. That's why it is important to train on quality data; things you can build on top of.

▲

parineum 3 days ago | parent [-]

What if you are trying to fix one of these things and needed a list of replacement parts?

	▲	esafak 3 days ago \| parent [-]
		Not the right problem for this model. Any RAG-backed SLM would do; the important part is being backed by a search engine, like https://google.com/ai

▲

DennisP 3 days ago | parent | prev [-]

Just because it's in the training data doesn't mean the model can remember it. The parameters total 60 gigabytes, there's only so much trivia that can fit in there so it has to do lossy compression.