is there a shorthand/heuristic to calculate pvalue given n samples and effect size?

There are no great shorthands, but here are a few rules of thumb I use:

- for N=100, worst case standard error of the mean is ~5% (it shrinks parabolically the further p gets from 50%)

- multiply by ~2 to go from standard error of the mean to 95% confidence interval

- scale sample size by sqrt(N)

So:

- N=100: +/- 10%

- N=1000: +/- 3%

- N=10000: +/- 1%

(And if comparing two independent distributions, multiply by sqrt(2). But if they’re measured on the same problems, then instead multiply by between 1 and sqrt(2) to account for them finding the same easy problems easy and hard problems hard - aka positive covariance.)