You are 100% correct, but this is still mostly fine: without SGX, you need to completely trust Signal, since it could trivially modify the server-side code. But with SGX, you only need to trust that Signal and Intel won't both collude.

The most likely attacks on Signal involve trusted insiders or configuration errors, and SGX mostly prevents these, since to exploit it, you'd need to bribe insiders in both Signal and Intel, or find configuration errors in both of their software stacks.

Collusion is certainly still possible, but it's much harder to pull off, since it typically requires nation-state-level resources to exploit. Signal does actually have nation-state adversaries, but the vast majority of other software projects don't.

(I personally think that remote attestation is the single biggest risk to the free software movement, but I begrudgingly accept that Signal is a very good use case for it.)