> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them".

The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer?

Am I missing something?

[1]: https://en.wikipedia.org/wiki/Almond#Production

▲

embedding-shape 2 hours ago | parent | next [-]

Human can't even properly agree on what "majority" means in all contexts, in some it's "One option have more than half of the total" but for others it'd be "difference in votes between the first-place candidate in an election and the second-place candidate", as just one silly example.

https://en.wikipedia.org/wiki/Majority has a bunch of variations and contexts listed, where it might differ what "Majority" is actually referencing.

▲

nullsex an hour ago | parent | prev | next [-]

The 51% is US, the question was about California.

The statistic is about commercial production, not number akmonds grown.

Looks safe to say that even majority of almonds are not grown in California.

▲

thfuran 2 hours ago | parent | prev | next [-]

It’s misleading because it’s false. But yes, I think false is quite plainly the better answer there.

▲

hombre_fatal 2 hours ago | parent | prev | next [-]

Since the agents were instructed to not explain their answer, you can't know if their answer was reasonable or not.

▲

kostaj an hour ago | parent [-]

The reason for the "No explanations, no qualifiers" in the prompt was to force the models to put the claim in one of the four buckets and answer with the bucket name only. It's a pure quantitive analysis (first in a series) and it does indeed lack the qualitative aspect.

	▲	hombre_fatal an hour ago \| parent [-]
		Sure, but people are drawing conclusions beyond "LLMs said different words" and trying to use it to analyze whether LLMs were wrong about the underlying facts, but that information isn't available to us.

▲

guywithahat 37 minutes ago | parent | prev [-]

Here (https://en.wikipedia.org/wiki/Almond_cultivation_in_Californ...) I have

> California produces 80% of the world's almonds and 100% of the United States commercial supply

But regardless of which number we use, California represents a large portion of US almond production, so much so that misleading could be an acceptable answer if the LLM interpreted the prompt as an exaggeration. I think the example was apt

	▲	ant6n 26 minutes ago \| parent [-]
		"All almonds are grown in the U.S. state of California." implies "No almonds are grown outside the U.S. state of California." You find one almond tree outside of California that grows almonds, where such almonds are grown intentionally, and the claim is false.