> Having an archive of "curated" training data seems like it is going to be important

the justification for not doing that is probably "prohibitively expensive given the amount of data involved". they'd need a bunch of human reviewers combing through massive troves of data. it's probably cheaper to "sort of fix" it after the fact.

> perhaps there's ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable)

as a lecturer once said to me about my idea for a masters dissertation project that would classify news sites based on right/left tendencies -- "that sounds dangerously political". especially given the current let's all shout at each other political climate.

aside: someone built this and it was a fully fledged company, which has always annoyed me.

▲

JKCalhoun 5 hours ago | parent [-]

"…they'd need a bunch of human reviewers combing through massive troves of data…"

Yeah, I concede that. It doesn't need to be done over night. Having a static repo of data though that you can work through over time (years)—removing some data, add pre-curated data to. In so many years you can have a pretty good "reference dataset".

	▲	gowld 5 hours ago \| parent [-]
		I think some of the thousands of people working on training LLMs have tried some of the low-hanging-fruit ideas we can brainstorm of the top of our head 5 years later.