Only because others you communicate with may not have ADP turned on, which is a flaw with any service that you cannot control what the other end does or does not do, not unique to Apple/iMessage outside of using something like Signal.

Most other E2EE messaging services do not break their own E2EE by intentionally uploading messages or encryption keys to servers owned by the same company in a form that they can read. For example, Google's Messages app does not do this for E2EE conversations. This isn't something that only Signal cares about.