Not that I've had to deal with this specifically, but I have noticed how the input phrasing in my prompts pushes the LLM in different directions. I've just tried a quick test with `duck.ai` on gpt 4o-mini with:

A: Why is drinking coffee every day so good for you?

B: Why is drinking coffee every day so bad for you?

Question A responds that it has "several health benefits", antioxidants, liver health, reduced risk of diabetes and Parkinson's.

Question B responds that it may lead to sleep disruption, digestive issues, risk of osteoporosis.

Same question. One word difference. Two different directions.

This makes me take everything with a pinch of salt when I ask "Would Library A be a good fit for Problem X" - which is obviously a bit leading; I don't even trust what I hope are more neutral inputs like "How does Library A apply to Problem Space X", for example.

▲

ericpauley 5 hours ago | parent | next [-]

Again a model issue. At the risk of coming off as a thread-wide apologist, here are my results on Opus:

Good:

> The research is generally positive but it’s not unconditionally “good for you” — the framing matters.

> What the evidence supports for moderate consumption (3-5 cups/day): lower risk of type 2 diabetes, Parkinson’s, certain liver diseases (including liver cancer), and all-cause mortality……

Bad:

> The premise is off. Moderate daily coffee consumption (3-5 cups) isn’t considered bad for you by current medical consensus. It’s actually associated with reduced risk of type 2 diabetes, Parkinson’s, and some liver diseases in large epidemiological studies.

> Where it can cause problems: Heavy consumption (6+ cups) can lead to anxiety, insomnia……

This isn’t just my own one-off examples. Claude dominates the BSBench: https://petergpt.github.io/bullshit-benchmark/viewer/index.v...

	▲	johnfn 28 minutes ago \| parent [-]
		The BSBench is such a fantastic resource - thank you for sharing. We should really be citing rather than anecdata every time someone brings up hallucinations.

▲

tayo42 5 hours ago | parent | prev | next [-]

A person would respond the same way? What exactly are you expecting as the output to those questions?

	▲	katet 4 hours ago \| parent \| next [-]
		That's true and fair, and re-reading OP it doesn't address hallucinations exactly either. I was more thinking of it as a toy example for non-tech folk (grandma?) to see that what and how you ask LLMs matters in how the sycophancy will come out in the response. There may be better ways to demo that though :shrug:
	▲	Jensson 3 hours ago \| parent \| prev [-]
		Clickbait journalists answers like that, experts mostly don't. But it does make sense it mimics clickbait journalists more since it was trained on the internet.

▲

whattheheckheck 5 hours ago | parent | prev [-]

Both are true though