I have a stupid benchmark: Bongard Problem 2 from here (https://matheminutes.blogspot.com/2019/04/bongard-eggs.html). It's an open-ended visual task that is easy enough for humans, but models are still somewhat bad at visuals, so it's one of the few remaining reliable ways to trigger hallucinations. This one answered correctly, but then felt the need to add a fun fact that was wrong:

> The rule separating the two baskets is layering (which color is on top). Left basket: In every pattern on all six eggs, the blue shape is on top of (overlapping) the green shape. Right basket: In every pattern on all six eggs, the green shape is on top of (overlapping) the blue shape. [true solution]

> (Note: Both baskets contain the exact same distribution of motif counts per egg—two eggs with 2 pairs, three eggs with 3 pairs, and one egg with 4 pairs—serving as a classic Bongard distractor to hide the occlusion rule.) [factually incorrect and also would make no sense as a distractor]

And then trying a harder puzzle just makes it bullshit completely.

I don't understand why Gemini models always add these "it's typical XYZ" bits, which are typically so much worse than everything else in the answer.

▲

rbobby a day ago | parent | next [-]

The sample problem I kind of gave up on and guessed sort of... it couldn't be sorted by color that's too obvious.

And then the real problem number 1. The answer is "Left - each figure contains an even number of arrow heads. Right - each figure contains an odd number of arrow heads"... but on the right I can clearly see that the three eggs across the top have an even number of arrowheads.

▲

mlmonkey a day ago | parent [-]

It's not the number of arrowheads on the egg; it's the number of arrowheads on each of the little figures on the egg. Those eggs you mention have 2 figures each, true, but each figure has just 1 arrowhead each.

	▲	rbobby 20 hours ago \| parent [-]
		Huh. Damn.

▲

nxtfari a day ago | parent | prev [-]

If you have an Ultra subscription (I don't or I would try it) I would be really curious if Carbon is able to solve your harder Bongard problem without making stuff up given it's absurdly high AA Omniscience non-hallucination score, it's supposed to be like 30%+ better than nearest non-Google competitor. Although Hydrogen xhigh apparently scored within 5% of Carbon so who knows ¯\_(ツ)_/¯

▲

Kotlopou a day ago | parent [-]

I don't have that subscription, but for the record, it's this problem: https://imgur.com/a/idyH0Kh

I won't share the solution (you have to keep some mystery in the world), but I showed this a year ago at a high-school reunion and five people independently found the same solution, so it should be well-posed.

▲

nxtfari a day ago | parent [-]

Ha, okay I caved and spent the 5¢ to ask Carbon the problem through openrouter and it was indeed able to solve it. No random hallucinated fact either. Thought you might want to know. Cool problem!

	▲	Kotlopou a day ago \| parent [-]
		Interesting indeed, thanks! Are you able to share the conversation? This problem was made to stump Gemini last December. I actually made another one back then, designed to be more brutal, and I'm curious whether Carbon figures that one out, if you have a few more cents to spare: https://substack-post-media.s3.amazonaws.com/public/images/e... A friend solved it in an hour with just a few vague hints (there's no need to Google anything; two subproblems require a bit of knowledge that a caveman would not have but you surely do; it's all one problem and it is neither necessary nor recommended to solve 12 subproblems). Back when we made this, we wondered with this friend whether it would still be possible to stump Gemini the next year. (Already Gemini 3 could solve a few straightforward Bongard problems, including at least one of the subproblems here.) Bongard problems are cool because you can create infinite variations of them on pretty much any concept, and they seem to capture some kind of flexibility in thinking (when they are well-designed -- there are definitely arbitrary-feeling ones), so this would be a pretty big milestone for me.