The orthogonal LoRA constraint is interesting. Have you thought about whether orthogonality conflict with the timestamped training? If two temporally adjacent observations should produce similar LoRA updates, orthogonality would actively push them apart. Maybe you want similarity for recency, orthogonality only for distinct episode types?

This is more of a conceptual idea from an AI hobbyist; hopefully, the big claims for motivation are not too distracting. After doing too many basic-level tutorials, this could be an interesting intermediate-level project to apply modern AI architectures. What is your opinion?