There's currently a big market for figuring out ways to measure intelligence. With a particular interest in ways that humans can score much higher than LLMs. If you have some ideas please do share!

▲

eli 5 hours ago | parent | next [-]

Why? Seems like benchmarks that closely mirror the tasks you'd want an LLM to help with would be a lot more useful than some general intelligence benchmark.

▲

ranyume 4 hours ago | parent | prev | next [-]

Give away access to the model and go ask people from time to time if the model was of use to the person and if they were able to make the model work with them.

▲

malfist 5 hours ago | parent | prev [-]

I am no where close to qualified to do that. Hell, experts can't even define what intelligence is, much less define a test for it

	▲	hyperhello 5 hours ago \| parent \| next [-]
		This is like arguing about whether a hot dog is a sandwich (of course it is) or whether the chicken or the egg was first (obviously the egg since all chickens come from eggs). Intelligence is just problem solving in the context of self-awareness. Machines don't have it and never will but they can simulate the process given inputs. You can argue whether humans and animals truly possess self-awareness and in what degree, but the definition of intelligence is as simple as the hot dog debate.
	▲	whattheheckheck 5 hours ago \| parent \| prev [-]
		It was defined in Animal Intelligence by George John Ramones in 1882 as "intelligence is the capacity to do the right thing at the right time. It is the ability to respond to the opportunities and challenges presented by a context"