Remix.run Logo
egberts1 3 days ago

Asking AI to stay true to my requested parameters is hard, THEY ALL DRIFT AWAY, RANDOMLY

When working on nftables syntax highlighters, I have 230 tokens, 2,500 state, and 50,000+ state transitions.

Some firm guidelines given to AI agents are:

1. Fully-deterministic LL(1) full syntax tree.

2. No use of Vim 'syntax keyword' statement

3. Use long group names in snake_case whose naming starts with 'nft_' prefix (avoids collision with other Vim namespaces)

4. For parts of the group names, use only nftables/src/parser_bison.y semantic action and token names as-is.

5. For each traversal down the syntax tree, append that non-terminal node name from parser_bison.y to its group names before using it.

With those 5 "simple" user-requested requirements, all AI agents drift away from at least each of the rules at seemingly random interval.

At the moment, it is dubiois to even trust the bit-length of each packet field.

Never mind their inability to construct a simple Vimscript.

I use AI agents mainly as documentation.

On the bright side, they are getting good at breaking down 'rule', 'chain_block stmt', and 'map_stmt_expr' (that '.' period we see at chaining header expressions together; just use the quoted words and paste in one of your nft rule statements.

legacynl 2 days ago | parent [-]

Im a dev (who knows nothing about nftables) and I don't understand your instructions. I think maybe you could improve your situation by formulating them as "when creating new groupnames use the semantic actions and token names as defined in parser_bison.y" I.e. with if conditions so that the correct rules apply to the correct situations. Because your rules are written as if to apply to every line of code, it might unnecessarily try to incorporate context even when it's not applicable.