What you are describing is a failure to integrate AI into said company systems. I have seen quite a few companies now that buy MS AI products with great hopes only to be severely disappointed, because they may as well have just used vanilla ChatGPT (in fact then they would at least get newer models faster). But there are counter examples too. If you can pull all your company documentation into a vector db and build a RAG based assistant, you can potentially save countless hours across your workforce and possibly customers too. But this is not easy and also requires some level of UI interactivity that noone really offers right now. In fact they can't offer it, because you usually need to integrate ancient, arcane sources into your system. So you do have to write a lot of integration code yourself at every step. Not many companies are willing to spend that kind of money and effort, because managers just want to buy a MS product and be done with improving efficiency by next quarter.

I have been using vector based RAG for about two years now, I am not knocking the tech, but last year I started experimenting with going way back in time and also in parallel trying BM25 search (or hybrid BM25 and vector). So: not even a very good example use case of LLMs, the tech is not always applicable.

EDIT: I am on a mobile device and don’t have a reference handy but there have been good papers on RAG scaling issues - basically the embedding space gets saturated (too many document chunks cluster in small areas of the embedding space), if my memory is correct.