I recently migrated from lazy.nvim to vim.pack and it was way easier than expected:

https://github.com/azemetre/dotfiles/pull/61/files

I followed the work of another neovimmer where he was able to replicate deferring with vim.pack. Brought my startup time down to sub 100ms.

Definitely worth it to me as it's one less "core" plugin to maintain. Having things like telescope or trouble are one thing, it's quite another to rely on a plugin that changes the way neovim interacts with loading.