I don’t feel like the sentence structure in English is particularly flexible. It relies on word order for things other languages prefer to express with cases, prepositions, or context.

I think the disparity between the written and spoken language is big enough of a problem to make the language difficult; certainly not less so than extra verb forms or gender in other languages. For example, I don’t find French remarkably more difficult than English, because even though it has both of those features, reading it is fairly straightforward, which makes the language so much more comfortable.

Another part of English that I personally always found challenging (and which still trips me up from time to time) are the verb tenses. Just wrapping my head around them and all the context switching to know which one to use for the past/present/future relative to what I’m talking about right now. Perhaps most people don’t feel as strongly about them, though.

All that said, I consider English among the easier languages to learn, but still quite challenging. Difficulty-wise, one can do far, far better.