I found it very difficult to parse your description, "Clear, not too long not too short prompt, for LLMs to go out and research freelance consulting gigs for one specific IT domain, and in one specific country in Europe, including maybe opportunities driven from temp agencies based in geographically close countries."

( I was trying to quote a single sentence and then realised it ran on for the whole paragraph. )

Given how difficult I found that to follow, are you sure your prompt is actually "Clear, not too long not too short"? We now only have your word for it. I too had assumed that was a prompt given to an LLM to further prompt agents.

▲ geon 5 hours ago | parent | next [-]

That was not the prompt.

	▲	xnorswap 5 hours ago \| parent [-]
		I know that. We don't know what the prompt was. We only have a self-assessment of the quality of the prompt from the person who wrote it. It sounds like they're hitting a data source quality issue, which is hardly uncommon in scraping. It's common for job boards to obscure who the real clients are, and if the scraping engine is LLM powered ( rather than LLM written ), then I would expect it to accidentally present agencies as the contracting organisation sometimes. Breaking down the process so you can inspect the messy middle of a data pipeline is an important part of software engineering, but it sounds like they've tossed a messy task at an LLM and expected it to be proficient end-to-end.

▲ tcp_handshaker 5 hours ago | parent | prev [-]

You can easily test this yourself with the SOTA models....or read the corroborating literature...

"General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778

"...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks..."

▲ xnorswap 5 hours ago | parent | next [-]

What proportion of the human population could answer the example from that paper?

    Question: Strangers A, B, C, D, and E line up from youngest on the left to oldest on the right. Their clothing
    colors and shoe colors all differ, and they come from five different regions.
    Known facts:
    1. A is from Morocco.
    2. D is five years older than B.
    3. E is older than A.
    4. C stands next to D.
    5. A stands next to B.
    6. The person in teal shoes is not adjacent to the person from Vanuatu.
    7. One twelve-year-old wears yellow shoes.
    8. The person in orange shoes wears white clothing.
    9. The person in blue clothing is from Chile.
    10. The youngest person wears red shoes.
    11. Counting from the right, the fourth person comes from South Africa.
    12. E wears yellow clothing.
    13. The person in green shoes does not wear multicolored clothing.
    14. Two people are twelve years old, ordered by birth month.
    15. One adult is thirty-five years old, and that age is sixteen less than the combined ages of the other four.
    If you multiply every possible age C might have, what number do you obtain?

What does "The person in green shoes does not wear multicolored clothing" even mean?

Nowhere is "multicoloured" defined, are we to assume it should be treated as a colour and implied that someone else must be wearing "multicoloured clothing"? Because strictly that doesn't logically follow, and it ought to be phrased as "The person in green shoes is not the person wearing multicoloured clothing" if that is the case.

This is an extremely hard logic puzzle, especially since it's revealed at the end that there are multiple solutions.

I'd expect anyone to struggle unless armed with prolog.

▲

defrost 5 hours ago | parent [-]

> What does [ 13 ] even mean?

It means that it is possible for someone to be wearing a white shirt and yellow pants (say), but the person in green shoes came from the set of a Wes Anderson film.

▲

xnorswap 4 hours ago | parent [-]

That's cute, but it's driving me crazy, I guess I'll have to sit down and solve it to figure out how it's meant to be clued, assuming it's not just a red-herring entirely.

	▲	xnorswap 4 hours ago \| parent [-]
		Right, this is based on pen-and-paper working, so I could be wrong, but I think it's a red-herring, the set of "clothing" seems to be: white, blue, yellow, and is otherwise undefined. But we know from [1], [2], [4], [5] and [11], that the order must be: A, B, C, D, E or A, B, E, D, C. Which makes C either the older 12 year old or the 35 year old. The key this is that D can't be a 12 year old without A being in slot 2, but A can't be in slot 2 because slot 2 is South Africa and A is Morocco. Trying to reach the shoes + Vanuatu clue is a complete waste of time, paying any attention to shoes or clothing is a waste of time, it feels like there ought to be a way to narrow it down to one of those two configurations, but the clothing is too ambiguous, the shoes end up irrelevant. What a frustrating puzzle, where half the clues are seemingly redundant.

▲ fn-mote 4 hours ago | parent | prev [-]

Super interesting, thanks for the reference.

I particularly found the note about “local collapse” helpful (near the end of section 3). The idea is that even though benchmarks contain a wide variety of different reasoning tasks, each individual problem requires only a few skills - unlike this benchmark where they deliberately construct tasks that span many categories.