Easy listening implies that there’s not much of anything there. Nothing surprising or unique about the song or the performance. No insightful message and nothing worth reflecting on after.

I don’t think the alternative is “difficult” for its own sake. Rather, those who would use the term as a pejorative are likely seeking new experiences and viewpoints in their music and get bored by same old diatonic melodies over plain inoffensive grooves. Novelty is a source of dopamine for some.

A lot of jazz music is difficult to the untrained ear, and I have distinct memories of hearing albums that I now feel are too conservative but in my youth thought they were too chaotic. I now understand that it was never difficult from the performer’s perspective - just high level musicians playing the music they hear. I wish everyone could hear jazz just once through the ears of a jazz musician.