He’s just as nice and fun in person as he seems online. He’s put time into using these tools but isn’t selling anything, so you can just enjoy the pelicans without thinking he’s thirsty for mass layoffs.

I know what you mean, but weighing up things:

- oh, it's that guy again

+ prodigiously writes and shares insights in the open

+ builds some awesome tools, free - llm cli, datasette

+ not trying to sell any vendor/model/service

On balance, the world would be better of with more simonw shaped people

he's incredibly nice and a passionate geek like the rest of us. he's just excited about what generative models could mean for people who like to build stuff. if you want a better understanding of what someone who co-created django is doing posting about this stuff, take a look at his blog post introducing django -- https://simonwillison.net/2005/Jul/17/django/

Because he writes a lot about it.

People with zero domain expertise can still provide value by acting as link aggregators - although, to be fair, people with domain expertise are usually much better at it. But some value is better than none.

Because he's prolific writer on the subject with a history of thoughtful content and contributions, including datasette and the useful Python llm CLI package.

For every new model he’s either added it to the llm tool, or he’s tested it on a pelican svg, so you see his comments a lot. He also pushes datasette all the time and I still don’t know what that thing is for.

Nah, we still treat people thinking about it as crackpots.

Honestly, getting into the whole AI alignment thing before it was hot[0], I imagined problems like Evil People building AI first, or just failing to align the AI enough before it was too late, and other obvious/standard scenarios. I don't think I thought of, even for a moment, the situation in which we're today: that alignment becomes a free-for-all battle at every scale.

After all, if you look at the general population (or at least the subset that's interested), what are the two[1] main meanings of "AI alignment"? I'd say:

1) The business and political issues where everyone argues in a way that lets them come up on top of the future regulations;

2) Means of censorship and vendor lock-in.

It's number 2) that turns this into a "free-for-all" - AI vendors trying to keep high level control over models they serve via APIs; third parties - everyone from Figma to Zapier to Windsurf and Cursor to those earbuds from TFA - trying to work around the limits of the AI vendors, while preventing unintended use by users and especially competitors, and then finally the general population that tries to jailbreak this stuff for fun and profit.

Feels like we're in big trouble now - how can we expect people to align future stronger AIs to not harm us, when right now "alignment" means "what the vendor upstream does to stop me from doing what I want to do"?

--

[0] - Binged on LessWrong a decade ago, basically.

[1] - The third one is, "the thing people in the same intellectual circles as Eliezer Yudkowsky and Nick Bostrom talked about for decades", but that's much less known; in fact, the world took the whole AI safety thing and ran with it in every possible direction, but still treat the people behind those ideas as crackpots. ¯\_(ツ)_/¯

There are good analogies to be had in mythologies and folklore, too! Before there was science fiction - hell, even before there was science - people still occasionally thought of these things[0]. There are stories of deities and demons and fantastical creatures that explore the same problems AI presents - entities with minds and drives different to ours, and often possessing some power over us.

The arguably most basic and well-known example are entities granting wishes. The genie in Alladin's lamp, or the Goldfish[1]; the Devil in Faust, or in Pan Twardowski[2]. Variants of those stories go in detail over things we now call "alignment problem", "mind projection fallacy", "orthogonality thesis", "principal-agent problems", "DWIM", and others. And that's just scratching the surface; there's tons more in all folklore.

Point being - there's actually decent amount of thought people put into these topics over the past couple millennia - it's just all labeled religion, or folklore, or fairytale. Eventually though, I think more people will make a connection. And then the AI will too.

--

As for current generative models getting spooky, there's something else going on as well; https://www.astralcodexten.com/p/the-claude-bliss-attractor has a hypothesis I agree with.

--

[0] - For what reason? I don't know. Maybe it was partially to operationalize their religious or spiritual beliefs? Or maybe the storytellers just got there by extrapolating an idea in a logical fashion, following it to its conclusion. (which is also what good sci-fi authors do).

I also think the moment people started inventing spirits or demons that are more powerful than humans in some, but not all ways, some people started figuring out how use those creatures for their own advantage - whether by taming or tricking them. I guess it's human nature - when we stop fearing something, we think of how to exploit it.

[1] - https://en.wikipedia.org/wiki/The_Tale_of_the_Fisherman_and_... - this is more of a central/eastern Europe thing.

[2] - https://en.wikipedia.org/wiki/Pan_Twardowski - AKA the "Polish Faust".

They've gotten quite good at reinventing the Torment Nexus

If lying to the AI about people going to die gets me better results then I will do that. Why shouldn't I?

Let's say I have a "true block on the bad thing". What if the prompt with the threat gives me 10% more usable results? Why should I never use that?

"100% foolproof" is not a realistic goal for any engineered system; what you are looking for is an acceptably low failure rate, not a zero failure rate.

"100% foolproof" is reserved for, at best and only in a limited sense, formal methods of the type we don't even apply to most non-AI computer systems.

If it’s anything like the AI expert systems I’ve heard about in insurance, it will be a tool that is optimized for low effort, but will be used carelessly by end users, which isn’t necessary the fault of the AI. In automated insurance claims adjustment, the AI writes a report to justify appealing patient care already approved by a human doctor that has already seen the patient in question, and then an actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope.

AI systems with a human in the loop are supposed to keep the AI and the decisions accountable, but it seems like it’s more of an accountability dodge, so that each party can blame the other with no one party actually bearing any responsibility because there is no penalty for failure or error to the system or its operators.