I saw on Twitter that in an ML course at Tsinghua University, one of the tests asks students to write quizzes that fail the most LLM models as possible.

What if we create a benchmark that works like this and assigns ELO scores? Models fight head-to-head by writing a question, a bug, or an incomplete implementation, which the opponent has to answer, fix, or finish.

▲

vincnetas 2 hours ago | parent | next [-]

We could call this "generative adversarial network" (GAN) :)

https://en.wikipedia.org/wiki/Generative_adversarial_network

▲

wwind123 an hour ago | parent [-]

This kind of approach would generally still need human guidance, otherwise these models might get stuck in weird niche corners of the problem space that would not be relevant to any real world project.

	▲	ben_w 14 minutes ago \| parent [-]
		We could call this "reinforcement learning from human feedback" (RLHF) :) https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...

▲

olmo23 an hour ago | parent | prev [-]

How do you prevent degenerate strategies? I could trivially give a model a SHA256 hash and ask it to provide the source input.

In class you'd probably want a rule saying at least one LLM should be able to figure out the answer, but in a head-to-head I'm not sure how to solve it.

	▲	an hour ago \| parent [-]
		[deleted]