Y'all are sleeping on custom lint rules.

Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you. Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that explains entire architectural principles, concurrency rules, etc. Stuff that is very tailored to your codebase and are far more precise than a generic compiler/lint error.

Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it".

To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if "is variable XYZ unused?" or "does variable X oveshadow Y" or other more "esoteric" lint rules will not catch it. Likelihood is high that the LLM just hallucinated some false positive lint rule anyways giving you a false sense of security.

Clever! Sharing my lightning test of this approach.

Context - I have a 200k+ LOC Python+React hobby project with a directory full of project-specific "guidelines for doing a good job" agent rules + skills.

Of course, agent rules are often ignored in whole or in part. So in practice those rules are often triggered in a review step pre-commit as a failsafe, rather than pulled in as context when the agent initially drafts the work.

I've only played for a few minutes, but converting some of these to custom lint rules looks quite promising!

Things like using my project's wrappers instead of direct calls to libs, preferences for logging/observability/testing, indicators of failure to follow optimistic update patterns, double-checking that frontend interface to specific capabilities are correctly guarded by owner/SKU access control…

Lots of use cases that aren't hard for an agent to accurately fix if pointed at directly, and now that pointing can happen inline to the agent work loop without intervention through normal lint cleanup, occurring earlier in the process (and faster) than is caught by tests. This doesn't replace testing or other best practices. It feels like an additive layer that speeds up agent iteration and improves implementation consistency.

Thanks for the tip!

I realized this recently and I've been creating a RuboCop plug-in[1] to automatically have the LLM code better match my personal style. I don't think it'll ever be perfect, but if it saves me from moving a few bits around or adding spacing I'd rather see then it's wroth. The fun part is I'm vibe coding it, since as long as the tests verify the rules then it doesn't really matter much how they work. As a result adding a new rule is pasting in LLM generated code followed by what I'd prefer it look like and asking it to add a rule.

[1]: https://github.com/tristandunn/rubocop-vibe/

I just discovered https://megalinter.io/

Ha, I just had the LLM create my first custom eslint rule yesterday and was thinking that I should make more.

I like this idea but I can’t think of a concrete example to ground it. Can anybody share a real example?