You might be interested in this article by soatok [0] which discusses OMEMO and XMPP. Soatok has many reservations but I think if you use the most recent OMEMO version I think it should probably be fine.

Also of interest, OpenMLS [1]

[0] https://soatok.blog/2024/08/04/against-xmppomemo/

[1] https://github.com/openmls/openmls