One problem with his survey -- The Lizardman Constant:

> Researchers have demonstrated repeatedly in human surveys the stylized fact that, far from being an oracle or gold standard, a certain small percentage of human responses will reliably be [nonsense]: “jokester” or “mischievous responders”, or more memorably, “lizardman constant” responders—respondents who give the wrong answer to simple questions.

> Below a certain percentage of responses, for sufficiently rare responses, much or all of responding humans may be lying, lazy, crazy, or maliciously responding and the responses are false. This systematic error seriously undermines attempts to study rare beliefs such as conspiracy theories, and puts bounds on how accurate any single survey can hope to be.

> For example, 4% of respondents may endorse the claim ‘lizard-people rule the earth’, 5% of atheists believe in God, a surprising number of adults believe you see by shooting beams from your eyes, and so on. This cautions us against taking survey results about extremely unusual people or traits too literally, or expecting perfectly accurate results, as given the lizardman constant and other crud factors, it is entirely possible that some or all of the outliers may just be the lizardman constant at work.

https://gwern.net/doc/sociology/survey/lizardman/abstract

Scott Alexander has issued many studies in his time and is surely aware of this phenomenon. He was very cautious even in this study to calibrate for this sort of noise; see the section about Michaels you know.