I wonder how many proponents of this idea know that it's been done before? See https://en.wikipedia.org/wiki/Canton_of_Glarus - Protestants and Catholics had separate governments and tribunals in the same (rather small...) canton in the 17th century.

The summary I read also mentioned that the death penalty required agreement from both groups - highlighting one of the obvious complexities of such a scheme.

edit: a bit more detail