Do you an idea of how well these models perform on set theory problems or more niche fields in mathematics? So the model would have to both understand a paper that’s not in its training data, and use this to write proofs.

This is all fairly niche stuff I'm trying it on (well, the first three problems anyway), so yes, it needs me to give it several papers that are not in its training data and use them to write proofs. I would expect my experiences to transfer to set theory problems as well.