Yes. All you need is a formal grammar that defines the syntax of the text being generated. The llama.cpp repo has a decent writeup on the way that grammar-constrained decoding works in general [0], and it includes a bit of a simplistic example for generating syntactically-correct C code [1]. No reason this couldn't be extended to other things -- I personally have run experiments with generating syntactically-correct SQL statements when doing text-to-sql [2], and I think it's pretty cool.

[0] - https://github.com/ggml-org/llama.cpp/tree/master/grammars [1] - https://github.com/ggml-org/llama.cpp/blob/master/grammars/c... [2] - https://gist.github.com/HanClinto/b853571136fd42f8e2bde4c99c...