Last time I did anything with a BERT, you had to train or fine-tune. Is that not still true?

For me the cool bit is that it's all in-context learning or whatever so you can use it in any domain with zero setup.

Maybe bert and co. could do all the same things before, but the way in which you use them is quite different and that helps a lot.

It depends on your usecase but the models do show general capabilities. check this model out.

https://huggingface.co/MoritzLaurer/deberta-v3-large-zerosho....