If the model is predicting syntactically incorrect tokens in code you probably have bigger issues