It was more or less treated as the least-pejorative way of saying 'non-magic-aware' (in a similar-ish sense to 'Gentile'), but it seems like there's no way to have at least a little bit of negative implication given what it's denoting, and there's absolutely a sense that most wizards and witches consider themselves superior to the muggles.

Whereas 'mudblood' was specifically a slur against those of mixed heritage.