I don't think we're talking about "unhealthy diet" solely in the sense of eating unhealthy foods. The real issue is calorie intake. If you are taking in an unhealthy amount of calories -- regardless of the source -- you are unlikely to be able to outrun that diet.

And you're kinda agreeing with that point: you eat fast food a few times a week, but you don't eat breakfast. So perhaps your caloric intake is still at a reasonable level for your body, regardless of the source of those calories.