Folks often use things like LoRAs for that.