But isn't this a good thing?

I always try to avoid learning by pattern, because it would make me less flexible when composing.

No, not at all (imho), because if you're starting from scratch and don't even know which notes make up common chords or progressions, how in the world will you figure out how to compose something that sounds good / makes sense?

It's important to know the notes on the keyboard, but it's far more important to know how they work together in harmonious patterns. Composing entirely by ear is an option, but you still need to transliterate what your mind is hearing into notes on a page (preferably more complex than just a single note melody).

I 100% get where you're coming from, but I think a grounding in theory helps a ton with composition (even if you don't know how to play a specific instrument).

Well everyone learns differently. My earliest "compositions" (if you can call them that) are from grade school whistling into an old tape recorder and then transcribing them on our upright piano. Many of my friends who are musicians could also compose based on their ability to sing LONG before they had any working knowledge of music theory.

Additionally let me call out that sight reading and composition are completely different objectives.

I also sincerely doubt playing random notes on the piano as part of a sight reading process would make you a better composer unless you're aiming to be the next Karlheinz Stockhausen.