I'm having a hard time imagining a case where you'd be worse off with Django (compared to whatever alternative you may have chosen) in the case where the happy path for the thing you're trying to do doesn't exist natively in Django. Either way you're still farming out that capability to custom code or a 3rd party library.

I guess if you write a lot of custom code into specific hooks that Django offers or use inheritance heavily it can start to hurt. But at the end of the day, it's just python code and you don't have to use abstractions that hurt you.