new | show | ask | jobs Github

eminence32 4 hours ago

Pelicans and bikes can be in the training set without them training for this specific benchmark.

▲

j_maffe 4 hours ago | parent [-]

Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?

▲

2 hours ago | parent | next [-]

[deleted]

▲

barrenko 3 hours ago | parent | prev | next [-]

Would it? Tongue in cheek.

▲

asasidh 3 hours ago | parent | prev [-]

and that is bad because ?

▲

program_whiz 3 hours ago | parent [-]

the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.

	▲	wasabi991011 3 hours ago \| parent \| next [-]
		I would urge you to re-read the blog post you are commenting on. It pretty clearly explains how it is an interesting test independently of "see[ing] if the models can effectively compose an image of a novel concept outside the training set".
	▲	cyanydeez 3 hours ago \| parent \| prev [-]
		it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.