> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.

Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).

▲

stared an hour ago | parent | next [-]

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".

It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.

Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.

That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).

▲

diseasedyak an hour ago | parent | prev | next [-]

Yah, prediction interval instead, right? (I'm still learning statistics)

Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.

	▲	spider-mario 11 minutes ago \| parent [-]
		Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution. Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that. 95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”. 95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ). Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.

▲

maCDzP an hour ago | parent | prev | next [-]

Thank you for these, coz I learned a lot! Great that they are open access.

▲

jnwatson 3 hours ago | parent | prev | next [-]

Mind blown. The more I read about statistics, the less I know.

	▲	exogenousdata 2 hours ago \| parent [-]
		“There are three kinds of lies: Lies, damned lies and statistics.” - Mark Twain (attributed but unsubstantiated to Benjamin Disraeli)

▲

fr2029 3 hours ago | parent | prev [-]

the 2nd derivate of shannon covariance of noise begs to differ

	▲	2 hours ago \| parent [-]
		[deleted]