SFU is nice because it gives everyone privacy. Neither you or ISP know who you are communicating with.

Otherwise if everything is trusted I love that we can have conversations without depending on others over the internet :)

There's nothing bad about SFU, particularly the version you wrote, which forms the basis of Livekit. It would be my first choice for supporting larger groups in Briefing anyway. If the traffic is E2EE, it doesn't matter if an SFU is involved. The critical part is the signalling, in my opinion. This is where the initial communication is established. In the current version of my app, whose source code is yet to be published, this can happen via an untrusted server.