| ▲ | bieganski 2 hours ago |
| I tried taking the test, but ultimately stopped at Question 4: There are two sisters, Amy who always speaks mistruths and Sam who always lies. :) |
|
| ▲ | sixhobbits an hour ago | parent | next [-] |
| This is intentional I think, the models assume it's the classic riddle and choose A even though it's actually much simpler if both will always lie |
| |
| ▲ | ModernMech 6 minutes ago | parent [-] | | Yeah, most models used to get very confused with my riddle: "You're on one side of a river with a boat and a goat. How can you get to the other side?" Leaving out the fox and the lettuce, they would usually hallucinate them and come up with a crazy method of taking the goat back and forth for no reason. Good models don't get fooled by this anymore. |
|
|
| ▲ | InsideOutSanta 2 hours ago | parent | prev | next [-] |
| I thought you were joking, but that's actually the fourth question. Then I thought maybe it was meant to be a trick question, but none of the possible answers make sense. Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers. |
| |
| ▲ | pvitz 2 hours ago | parent | next [-] | | The trick here is that the classic version has one sister telling the truth and the other the lie. Then you would ask what the other sister would answer. In this case, you can just ask which path to take and then you take the other one. | |
| ▲ | dist-epoch an hour ago | parent | prev | next [-] | | It's a simple one. There are 2 paths. You ask C) "What path leads to the treasure". Both of them lie, so you pick the other path. | | | |
| ▲ | unglaublich 2 hours ago | parent | prev [-] | | And then you click one and it'll confidently say that another one is correct. The one later with the car and the glove is also bullshit. | | |
| ▲ | MatekCopatek an hour ago | parent | next [-] | | I thought it made perfect sense, I mean, they're intentional trick questions. They're meant to illuminate the fact that an LLM will produce a solution that is typical for the given assignment shape and the one with the car and the glove is a classic "objects moving in various directions with various speeds relative to eachother". But when a human reads it, they understand a glove dropping from a car in the middle of a bridge with sidewalks on each side will most probably just stay there, as it has no reason to move. | |
| ▲ | jabron 40 minutes ago | parent | prev [-] | | Yes, because another one is correct. Are you a frustrated LLM by chance? |
|
|
|
| ▲ | dist-epoch an hour ago | parent | prev | next [-] |
| The questions are fun, I got 10/10 correct, but you need to pay attention and not rush. All the questions have the same pattern - irrelevant extra details meant to trick the model into thinking they are important and should be used in the solution (Chekhov's gun), and one hidden "common sense" assumption - the frying pot will melt the ice cubes, the juggling balls will both fall to the ground, the glove drops and stays put. Arguably you could nit-pick most of the questions, "technically the friend could be a psychopath, ...", these are not math questions with a "formally provable answer". |
| |
| ▲ | NitpickLawyer an hour ago | parent [-] | | I answered all 10 but nothing happened, no score or anything. Is the score 10/10 if all answers are green when you click them? | | |
|
|
| ▲ | xqcgrek2 an hour ago | parent | prev [-] |
| Every one of the reply comments so far has missed the part where Sam is the one who always lies. Whoosh. |
| |