Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?

Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.

This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.

▲

malisper 2 hours ago | parent [-]

Yeah, that was a mistake. We should have waited until it was more complete before submitting it.

We were just really excited about what we had built and how fast it was that we wanted to submit it.

▲

sondr3 an hour ago | parent [-]

This absolutely does not pass any smell test, by your own admission you included the test files "by accident", could only barely pass the minimum of SQL for the benchmarks, included environment variables for "internal benchmarking" and a preview version you cannot immediately release because it might contain sensitive data? What kind of sensitive data are you putting in a Postgres slop fork in such a way that you cannot immediately release it, besides obvious admissions of cheating? This all just reeks.

	▲	malisper 36 minutes ago \| parent \| next [-]
		> This absolutely does not pass any smell test, by your own admission you included the test files "by accident" If you don't believe you can run the binary yourself. You'll see the numbers match the ClickBench numbers and that those files are dead code during the benchmark
	▲	setr 35 minutes ago \| parent \| prev [-]
		I mean, if the now-released open source variant performs roughly the same, and has no weird behavior in it… the matter is resolved, no? They’ve already submitted the best possible corrective step for any sus behavior they previously did. It doesn’t even really matter if the preview binary was in fact cheating, if the beta binary meets the benchmark results always. It’d be like cheating on a test you were gonna score 100% on anyways.