The newest submissions to Hutter Prize made crazy jumps by just replacing the LSTM with a transformer and training it.

35'000$ for a prompt and some training is a nice chunk of cash.

Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.