I just built one of these as well. For your Xbox controller, see if this works: find any Windows PC and download the Xbox Accessories app. Connect the controller (via USB) and update its firmware. Once I did this, I was able to pair it with the framework desktop via bluetooth (under linux) reliably, and it's been rock solid ever since. Apparently some of the models shipped with buggy firmware that linux really doesn't like for whatever reason.