As you noted briefly, a big drawback is not getting to take advantage of the cache. Seems like a pretty big drawback.

Yes, it will destroy most of the caching potential. On the other hand, the average context window needed to achieve the same type of task may be much smaller. This might make up for it. And with a better harness, fewer rounds may be needed. Plus, hopefully costs will go down. There is a lot of hope in this comment though.