It depends on your parasympathetic nervous system tone. After getting a fitness tracker that analyzes sleep and heart rate variability I've discovered that my body is rather sensitive to alcohol. I need to have a baseline level of cardiovascular endurance and be properly hydrated with good electrolyte balance to be able to truly enjoy two drinks. If I'm not doing moderate-intensity cardio five times a week and taking my supplements, then even moderate boozing will tank my HRV and negatively affect my sleep. Bad sleep means diminished cognitive performance and lethargy to the point that I'm not giving the world the best version of myself. To answer your question: yes, if I'm having two drinks a night and not taking care of myself I'd consider that to be a problem. A lot of people are probably in a similar boat but don't have the data to spot it so they just keep wearing themselves down in a half-tired haze.

I've also discovered that brown liquors are twice as bad for me in this regard so I now try to stick to wine, sake, or gin. I'm not mad at this as I've grown rather tired of what bourbon culture has seemingly become... the capitalization, enshittification and marketing games have largely put me off from the category.