If you benchmark is to be better than a human familiar with the task at hand, then you will be disappointed.

If you are not expecting them to be as good as people then their failures seem wholly unremarkable. The specific nature of tbe failures can be quite interesting in fact. They can expose deficiencies in the architecrure, training data, or presumptiona of those controlling the models.

In my experience very few people are claiming the capabulities that would be required for me to expect them to produce better than they currenrly are.

I'm not bothered by a fish's inability to quote Shakespeare.

▲

runarberg an hour ago | parent [-]

> I'm not bothered by a fish's inability to quote Shakespeare.

But I am bothered when they put a pencil in the mouth of a fish and tell us that this is the future of literature, that this qualifies as playwright, and that theaters will soon begin putting on plays written by a fish with a pencil in its mouth.

▲

Lerc an hour ago | parent [-]

But they are not claiming that current models are better than humans but later developments might.

Do you think that a descendant of a fish could be a playwright or put on a play?

▲

TehCorwiz 44 minutes ago | parent [-]

Objectively no. Fish form animals have been around far longer than mammals and we've touched the moon.

	▲	vhcr 7 minutes ago \| parent [-]
		Humans are fish.