Very hard to understand the meat behind all the fluff of the article, especially as the simulation code is not available, and as the presented simulated and original graphs are effectively the same (I don't see a disagreement).

It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.

And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.

▲

card_zero 6 hours ago | parent | next [-]

Huh? The point is that the two graphs come out looking the same, making the original no more meaningful than random.

Source code is here: https://github.com/pem725/Dunning-Kruger (found here: https://pem725.github.io)

	▲	5555watch 6 hours ago \| parent \| next [-]
		Thanks for finding the code! Now it's much more clear. The simulated data tries generating the true relationship between actual and perceived scores from 0.0 to 1.0, and bias in self-reporting from 0% to 100%. So the output graph should be the average of all these data generating processes, yielding perceived relationship around 0.5 and bias around 50%, with some high variation. If you have the access, run their Shiny code with these values, and you will see the published plot. I'd argue that this demonstration is much weaker than "making original no more meaningful than random". It's more that the "simulated 50% bias and 0.5 true correlation looks similar to what DK published", which is also far fetched given the data generation they did. Note: true random (what they were going for) would cover negative relationships, yielding the random true relationship around 0; and if they wouldn't correct the sign of Bias, it would also average at around 0; yielding a realistic "random" with the slope hovering about 50% for any percentile.
	▲	rawgabbit 3 hours ago \| parent \| prev \| next [-]
		Line 272 of SimDK.R is where they produced the quartiles for top and bottom performers. But their code for generating both is exactly the same. Code> # Lowest performers t.low.t <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$statistic # first quartile t-test (paired) t.low.p <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$p.value # first quartile t-test (paired) # Highest performers t.hi.t <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$statistic # first quartile t-test (paired) t.hi.p <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$p.value # first quartile t-test (paired)
	▲	oulipo 6 hours ago \| parent \| prev [-]
		Except "two graphs look the same" in no way means "therefore the results are equivalent" The article is baseless and fluffy

▲

jszymborski 6 hours ago | parent | prev [-]

It's annoying they didn't plot the new and old "perceived" curves on the same figure, but if you pay attention to the y-axis, there is a very big difference.

In the old plot, the bottom quartile has about a 50 percentage point margin between actual and perceived performance while the new one is 30 percentage points, which is a 50% difference between the old and new curve. The second quartile has 3x more margin in the old version relative to the new one.

▲

5555watch 6 hours ago | parent [-]

So what does that change? If there's no error bars on the graphs you can't discuss significant differences easily. And if they do differ, plotting differences will yield the U shape curve.

	▲	jszymborski 38 minutes ago \| parent [-]
		If the error bars are so large that the old and new analyses aren't statistically different, then the margins are too uncertain to make any claim in the original work... which is sorta the claim that the new work is making.