Author here. Let me know if you have any questions about the post or about pgrust.

Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust and postgres (see the proofs directory if you're curious). For cases where formal verification is not easy, we've taken the c implementation of a function and the rust implementation of a function and ran millions of inputs through each of them and confirmed they gave the same results every time.

We've only covered about 15% of the surface area so far, but in the process, we've discovered ~100 bugs in pgrust and ~20 bugs in Postgres itself. My favorite postgres bug we found is this one[0]. Postgres has a quadtree implementation. Due to floating point rounding, it was possible for a point to be neither above, nor below, nor even with the center point of the quadtree.

We've also entered engagements with Antithesis[1] to do Jepsen style fault testing and Aretta[2] to do more serious formal verification.

If you want to support the project, the easiest way is to give us a star on GitHub[3]

[0] https://www.postgresql.org/message-id/19597-39c532e61d78dff6...

[1] https://antithesis.com/

[2] https://aretta.ai/

[3] https://github.com/malisper/pgrust

▲

marginalia_nu 2 hours ago | parent | next [-]

How do you know if you're making the right optimizations?

I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but that only sometimes makes the production profiler sample move its needle.

e.g. I've recently been experimenting with using a cursed hybrid model in Marginalia's index, where based on a mincore probe, I switch between mmap and io_uring for reading a cluster of pseudoadjacent data. There are real tanglible benefits both in the test machine and in prod with this, but the numbers do not agree at all about how the needle moves :P

▲

malisper 2 hours ago | parent [-]

I would probably dig into the reasons for the differences in the benefit on the test machine and in prod

I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place

If you can consistently reproduce the performance difference, you're already half way there

	▲	marginalia_nu 2 hours ago \| parent [-]
		I'm pretty sure the reason for the difference is that production machine exists in a state of mixed memory residency and low grade resource contention that is incredibly hard to replicate in a test scenario (as the moment you start making queries the pages warm up, and the test becomes unreliable). The hard part about optimizing this type of code, IMO, is that there are so many cache layers, both in the CPU and the OS and sometimes in the storage medium. You can warm all of those caches up, but then you're testing a nonsense scenario that will basically never happen in a realistic scenario, where wall clock time is what matters, and not much the CPU is working or how many IOPS you're pushing.

▲

jnwatson 4 hours ago | parent | prev | next [-]

The floating point comparison bug is nightmare fuel. I could look at that for years and never spot the mistake.

	▲	wffurr 2 hours ago \| parent \| next [-]
		Fuzzers are brilliant at this and produce all kinds of insane floating point inputs.
	▲	bee_rider 3 hours ago \| parent \| prev [-]
		On the bright side it could probably run for years without hitting the mistake as well. But it is nice to get it out of there.

▲

throwaway7783 3 hours ago | parent | prev | next [-]

This is a great project. Thank you!

A question on 20s postgresql time - It does not look like you are accounting for reading data from disk? Wouldn't the aggregation query have to load data from disk first? Or is it somewhat guaranteed that the table is already in memory? The Rust version is clearly in memory (I am no rust expert, so that may not even be actually in memory, if its a generator).

	▲	malisper 3 hours ago \| parent [-]
		> A question on 20s postgresql time - It does not look like you are accounting for reading data from disk I choose the data size so that it would fit in memory on the machine I was testing on. fwiw, there's still a ton of overhead Postgres has that the toy example does not. For example Postgres will serialize the numbers into tuples and need to deserialize them to execute the query. That's why it's not an apples-to-apples comparison

▲

btown 4 hours ago | parent | prev | next [-]

If someone wanted to use this as a real-time WAL-tracking read-only mirror of a live production database, for analytics work, is it ready for that use case yet?

	▲	malisper 4 hours ago \| parent [-]
		You can try it. We're happy to help you with it, but expect there to be issues to work through. You would want to do it for something non-critical

▲

lizimo 4 hours ago | parent | prev | next [-]

Is `pgrcolumnar` the default storage layout for tables? It would be cool if the same storage engine outperforms vanilla Postgres under both OLTP and OLAP workloads.

AlloyDB from Google Cloud uses columnar storage like a secondary index, while the relations are still stored in TOAST.

	▲	malisper 4 hours ago \| parent [-]
		pgrcolumnar is not the default storage method. Right now, it's exposed as a table access method. There's lots of design space for how to do this so I want to avoid pre-committing to anything

▲

andriy_koval 3 hours ago | parent | prev | next [-]

what is your vision of this project? Do you think pgrust will eventually be prod ready?

	▲	malisper 3 hours ago \| parent [-]
		I want to build the best database possible. While Postgres is great, there are a lot of core issues that have been around for over a decade. We're working hard to get pgrust production-ready, and it will definitely be production-ready in the near future. I wouldn't be putting hundreds of thousands of dollars into this project if I didn't think we could build a production-ready database.

▲

doctorpangloss 4 hours ago | parent | prev [-]

“Show me the prompt.”