| ▲ | foxglacier a day ago | |
Yes. He's asking it to do something impossible then grading the responses - which must always be wrong - according to his own made-up metric. Somehow a program to help him debug it is a good answer despite him specifying that he wanted it to fix the error. So that's ignoring his instructions just as much as the answer that simply tells him what's wrong, but the "worst" answer actually followed his instructions and wrote completed code to fix the error. I think he has two contradictory expectations of LLMs: 1) Take his instructions literally, no matter how ridiculous they are. 2) Be helpful and second guess his intentions. | ||
| ▲ | Leynos a day ago | parent [-] | |
It's the following that is problematic: "I asked each of them to fix the error, specifying that I wanted completed code only, without commentary." GPT-5 has been trained to adhere to instructions more strictly than GPT-4. If it is given nonsense or contradictory instructions, it is a known issue that it will produce unereliable results. A more realistic scenario would have been for him to have requested a plan or proposal as to how the model might fix the problem. | ||