It's not tough if you do your job and your management has no issues with you. What others are doing doesn't affect your performance.