I'd personally got specific contract carveouts for this to only apply e.g. during working hours on company equipment (or even more liberal).
The GitHub liberal IP agreement is a good example of being even more chill here.