>"Is there a reason you don't throw in something like cross-attention on an embedding of the sequence description/data labels?"

Won't you need to attribute a weight to the data coming from the model and the weight for your data?