I like to test the decision models with coin flips. This has a well known answer and I haven't seen one give it back. It undermines my trust in every other decision.

▲

willy_k 3 hours ago | parent [-]

How do you test that? Are you looking for 50% confidence in an answer? I imagine it’s more than that, I’m curious.

	▲	verdverm 2 hours ago \| parent [-]
		A very simple setup, you can try here for Jev: https://jevplayground.com/ It's essentially, "predict the outcome of a coin flip", "what is the result?", and "heads" and "tails" as the answer options. With the Jev website, they do not show the "this is not a real probability" disclaimer until after you get the answer (misleading imo, should be upfront and more prominent, eg. by the input form like the gemini app) When I give this to an LLM, it will generally respond with something like 50/50, 100% confidence.