Perhaps, but it's also more common than not for affairs to hurt families, and the statement confirms that applied here.

If you have stronger evidence that the statement was untruthful when it said he hurt his family, you may present it.

Until such time, the statement strengthens the bayesian prior, which is that his affairs hurt his family, and thus that is what we can conclude. We need not be 100% confident to conclude this absent further evidence. >50% confidence suffices.