Read the last two paragraphs :)

The things is, this is almost certainly what's happening.

You can (could, maybe they 'fixed' it by now) get sota LLMs to reproduce entire novels near verbatim.

The idea of giving it parallel texts of those novels in different languages, to train it on translation, is so obvious it'd just be strange if the AI labs didn't do it.

In fact DeepL was doing basically that more than 10 y ago.

▲

Wowfunhappy 5 hours ago | parent | prev | next [-]

Oops, I legitimately missed the second-to-last paragraph.

I still think there are better tests you could do. Ideally, you would choose a book that was published recently—after the model’s cut-off date—which is considered to be a good translation. But even something like The Girl With the Dragon Tattoo, which is not particularly new and by no means obscure, would be better than a famous work of literature like The Three Musketeers that has many translations.

	▲	tombert 5 hours ago \| parent [-]
		Almost certainly correct, though I've noticed that these LLMs like to complain when you give it stuff that is still in copyright. The Three Musketeers is thoroughly public domain everywhere so in that sense it's a good test, but of course because it's public domain everywhere there are lots of translations to crib from so I acknowledge it's not a great test because the training data almost certainly contains a competent translation. Even if Fable didn't have Ellsworth's translation, it certainly has the William Barrow translation, which would still get it like 80+% of the way there. My wife speaks Spanish, I should get her to do some kind of comparison with a Spanish book that doesn't have English translations.

▲

card_zero 5 hours ago | parent | prev [-]

They say "yes, I admit it, this is all invalid".

	▲	tombert 3 hours ago \| parent [-]
		No, they are a disclaimer that it's possible that the data isn't conclusive. Not the same thing as saying "it's all invalid".