Dfu-util is one of those "user space drivers", so if you have a nonstandard protocol you'd have to add it directly to dfu-util. There's no intermediate API.

It's not easy to set up a fake or "remapped" USB device on most OS as far as I'm aware, if you were trying to write an adapter program that modified USB packets.