The root of this subthread was this claim that I made and you questioned:

> Teams that decide to delegate security responsibilities to AI are more likely to do things fast and loose in general.

Note the word delegate. I claimed that teams that delegate security responsibilities to AI are more likely to play fast and loose in general. That’s because delegating security to AI—not supplementing existing security practices with AI—is likely to be a good way to launch insecure garbage into the world. AI simply isn’t good enough to get security right on its own. Maybe someday it will be good enough, but like I wrote earlier, it ain’t there yet. And any team that plays fast and loose with security is likely to play fast and loose in general.

See any problems with that logic?

I only used vibe coding as an obvious example that shows there are lots of teams that delegate security responsibilities to AI. (Vibe coders are delegating almost everything to AI.)

> If you're a "regular person" vibe coder, you're not doing code reviews with a team anyway. You presumably had no teacher and no one to tell you your mistakes. So having a security bot is strictly an improvement.

How is it strictly an improvement? Before vibe coding, “regular people” couldn't launch insecure garbage upon an unsuspecting world—they couldn't launch anything. Do you believe that it’s "strictly better" that now everyone can launch insecure garbage courtesy of their AI minions? Do you think it’s “strictly better” that lots of users are having their data sucked into insecure apps and web sites that are destined to be hacked?

> Also I'm still unclear on why you think the explanation is crappy.

It’s crappy because it tells you how to use a tool (a custom SQL interpolator) without helping you understand the cause of the problem that the tool is trying to solve. You could read this ChatGPT explanation about avoiding SQL injection in Scala and not be any wiser about how to avoid that problem in other programming languages.

Worse, you would never learn from this explanation that the underlying cause of SQL injection is the same as for cross-site-scripting holes and a host of other logic and security problems in software. That’s because ChatGTP was trained on explanations of these problems scraped from the internet, and 99% of those explanations are superficial because the people who wrote them didn’t understand the underlying issues.

But if you deeply understand the following, you will never make this kind of mistake again in any programming language:

1. Every string is drawn from an underlying language and must conform to the syntax and semantics of that language.

2. To combine strings safely, you must ensure that they are all drawn from the same language and are combined according to that language’s syntax and semantics.

Therefore, as a programmer, you must (a) understand the language beneath each and every string, (b) combine strings only when you can prove that they have the same underlying language, and (c) combine strings only according to that underlying language’s syntax and semantics.

If you understand these things, you will know how to avoid all SQL injection and XSS holes and related problems in all programming languages. Things like escaping will make sense: it converts a string in one language into its equivalent string in another language. Further, you will know when you can safely delegate some of your responsibilities to tools such as parsers, type systems, custom SQL interpolators, and the like.

But you wouldn’t learn any of this from the ChatGPT explanation you received. Worse, you wouldn’t even think to ask for this deeper explanation because you would have no reason to suspect from ChatGPT’s explanation that this deeper explanation even existed.

In any case, I appreciate your willingness to continue this conversation. It’s been fun and educational and has forced me to articulate some of my ideas more clearly. Thanks!

▲

ndriscoll a day ago | parent [-]

But I delegate checks to tools all the time. e.g. I could spend my time checking whether locks are all used correctly in our code, or I could use libraries designed to force correctness[0]. An LLM isn't an ideal solution to linting, but if you're stuck with a language with a weak type system maybe that's all you can reasonably do.

The actual problem is that you're using strings at all. The SQL solution (that the scala macros do) is to use prepared statements and bound parameters, not to escape the string substitution. Basically, work in the domain, not in the serialized representation (strings). Likewise with XSS, you put the stuff into a Text node or whatever and work directly with the DOM so the structural interpretation has all already happened before the user data is examined.

But "work in the domain as much as possible" is a good idea for a whole bunch of reasons (as chatgpt said).

It did also several times indicate there was more to the story. It didn't just say "because that way is safer"; it said it

> Builds a structured query object, not a raw string

> Parameterizes inputs safely (turns $id into ? + bound parameter)

> Often adds compile-time or runtime checks

> Instead of manipulating strings, you’re working with a query AST / fragment system

And concluded by saying there's absolutely more detail, and that it's important to understand:

> If you tell me which library you’re using (Doobie, Slick, Quill, etc.), I can show exactly what guarantees sql"..." gives in your stack—those details matter quite a bit.

On vibe coded "garbage", I expect there won't be much of a market for such things (why would there be when you can also just vibe it?), so it will more be a personal computing improvement, which already limits the blast radius (and maybe already improves the situation vs the precarious-by-default SaaS/cloud proliferation today even with poor security). I also think tooling and vibe security will be better than median professional level by the time it's actually as easy as people claim it is to vibe code an application anyway. i.e. security (which is an active area of improvement to sell to professionals) will probably be "solved" before ease-of-use. Partly exactly because issues like code injection are already "solved" in better programming languages (which are also more concise and have better tooling/libraries in general), so the bot just needs to default to those languages.

[0] https://news.ycombinator.com/item?id=47693559

	▲	tmoertel a day ago \| parent [-]
		> But I delegate checks to tools all the time. e.g. I could spend my time checking whether locks are all used correctly in our code, or I could use libraries designed to force correctness[0]. Do you believe that because you can delegate some responsibilities without sacrificing important requirements that it follows that you can delegate all responsibilities without sacrificing important requirements? Do you not understand the difference between delegating to the computer proofs such as type checking that the computer can discharge faithfully without error and delegating something as wide and perilous as security to something as currently flawed as AI? > An LLM isn't an ideal solution to linting, but if you're stuck with a language with a weak type system maybe that's all you can reasonably do. No, in such a situation you can add LLM-based checks to your responsibility for security. But you can’t delegate away your responsibility to LLMs and say that you care about security. AI ain’t there yet. > The actual problem is that you're using strings at all. What percentage of the world’s existing code do you believe does not use strings at all? Tragically, that is the world we live in. > Basically, work in the domain, not in the serialized representation (strings). Sure, but you can’t do all your work in the domain. At some point you must take data from the outside world as input or emit data as output. And, even if you are lucky enough to work in a domain where someone has done the parsing and serialization and modeling work for you so that you have the luxury of a semantic model to work with instead of strings, who had to write that domain library? What rules did that person have to know to write that library without introducing security holes? > [ChatGPT] did also several times indicate there was more to the story. Great. Then show me how a person who didn’t know of the existence of the rules I shared with you in my previous post would naturally arrive at them by continuing your conversation with ChatGPT. > security (which is an active area of improvement to sell to professionals) will probably be "solved" before ease-of-use. I think that this is a naive hope. Security is different from virtually all other responsibilities in computing, such as ease of use, because getting it right 99.99% of the time isn’t good enough. In security, there is no “happy path”: it takes just one vulnerability to thoroughly sink a system. Security is also different because you must expect that adversaries exist who will search unceasingly for vulnerabilities, and they will use increasingly novel and clever methods. Users won’t probe your system looking for ease-of-use failures in the UI. So if you think that AIs are going to get security right before ease-of-use, I think you are likely to be mistaken.