AFAIK there is no way to share a GPU between a host and multiple VMs without commercial hardware

Thats the thing, LXC isn't a VM, its a container, so they all share the same kernel. I think its worth a shot, so I may test this.