I had the same idea, but now I a Postgres database that has very high latency for simple queries because the CPU is busy building large HNSW indexes.

My impression is that it might be best to do vector index construction separately from the rest of the data, for performance reasons. It seems vector indexes are several orders of magnitude more compute intensive than most other database operations.