ondemand is "just" a web frontend for using a traditional HPC cluster, which of course means its architecture is deeply cursed: https://osc.github.io/ood-documentation/latest/architecture....

Yeah, OOD is a giant RoR webapp; you need to be running it on a node that can submit to your cluster.