Say you were interviewing a human, to see how capable they were. You are allowed to give them take home work. What kind of questions would you ask, or tasks would you give, to try to get a measure of their competence? If you gave them a task, would you iterate with them on the design, or would you see what they could produce on their own, without input?

Measuring "intelligence" is hard, but giving an "intelligent" entity tasks, and seeing what comes out, and then comparing the output with others, seems like a very reasonable, relative, way to do it.

▲

CharlesW an hour ago | parent [-]

> Say you were interviewing a human, to see how capable they were.

Even if this was a good idea when applied to humans (it's not), LLMs aren't humans, and I worry about people who don't understand the difference.

	▲	nomel an hour ago \| parent [-]
		> Even if this was a good idea when applied to humans (it's not) I'm not sure I understand. What's not a good idea? I'm asking you how you would do it, with some possible examples. Or, are you saying it's a bad idea to try to measure how competent someone is before hiring them? > LLMs aren't humans, Not sure how this is relevant. My question was how to measure competence and "intelligence" for a task an entity, intelligent enough to do that task, will do. LLMs are not humans, but are usually used to complete tasks humans want completed that would usually be done by humans. That's where the most token spend is for them. Since that's what people are using them for, it seems reasonable to try to measure competence in those tasks.