Can the same mechanism be used to make sure it outputs syntactically correct code?

HanClinto 2 hours ago | parent | next [-]

Yes. All you need is a formal grammar that defines the syntax of the text being generated. The llama.cpp repo has a decent writeup on the way that grammar-constrained decoding works in general [0], and it includes a bit of a simplistic example for generating syntactically-correct C code [1]. No reason this couldn't be extended to other things -- I personally have run experiments with generating syntactically-correct SQL statements when doing text-to-sql [2], and I think it's pretty cool.

[0] - https://github.com/ggml-org/llama.cpp/tree/master/grammars [1] - https://github.com/ggml-org/llama.cpp/blob/master/grammars/c... [2] - https://gist.github.com/HanClinto/b853571136fd42f8e2bde4c99c...

▲

mmastrac 3 hours ago | parent | prev | next [-]

Yes, absolutely. The problem is integrating it into the inference engine. Not impossible, just _very hard_.

	▲	Majromax 3 hours ago \| parent \| next [-]
		Also less useful than one might think at first. Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify. Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.
	▲	3 hours ago \| parent \| prev [-]
		[deleted]

▲

wolttam 3 hours ago | parent | prev | next [-]

If the model is predicting syntactically incorrect tokens in code you probably have bigger issues

▲

vonneumannstan 3 hours ago | parent | prev [-]

Is this even a real issue? Are you consistently getting code out of frontier models + harnesses(Codex, Claude Code, etc) that have syntax errors?