Ha, that brings me back trying to make Fabric.js somehow usable for a 3D viewport at a startup I was working at a while ago. (We used esm.js instead of WASM for the 3d engine, to give you an idea)

It was a lot of wrangling and unexpected behavior, and we ultimately ended up implementing a more traditional 3d gizmo - though the application was also really only meant to be used at a desktop with a mouse instead of also supporting touch controls.