Using a built-in vector extension is convenient if you want to integrate vector similarity (“semantic search”) with faceted search. Some vector stores (e.g., qdrant) support vector attributes that can be matched against, though.

As mentioned by another comment, an advantage of using a separate vector store (on different hardware) is that (re-)building vector indices can cause high CPU load and therefore latency for regular queries to go up.