When you benchmarked Postgres did you disable WAL for the cache table? That may minimize the difference.

Unlogged table, yes. 1ms is more than fast enough so I didn't bother to look further.