Remix.run Logo
clarionbell 11 hours ago

Anyone with a decent grasp of how this technology works, and a healthy inclination to skepticism, was not awed by Moltbook.

Putting aside how incredibly easy it is to set up an agent, or several, to create impressive looking discussion there, simply by putting the right story hooks in their prompts. The whole thing is a security nightmare.

People are setting agents up, giving them access to secrets, payment details, keys to the kingdom. Then they hook them to the internet, plugging in services and tools, with no vetting or accountability. And since that is not enough, now the put them in roleplaying sandbox, because that's what this is, and let them run wild.

Prompt injections are hilariously simple. I'd say the most difficult part is to find a target that can actually deliver some value. Moltbook largely solved this problem, because these agents are relatively likely to have access to valuable things, and now you can hit many of them, at the same time.

I won't even go into how wasteful this whole, social media for agents, thing is.

In general, bots writing each other on mock reddit, isn't something the loose sleep over. The moment agents start sharing their embeddings, not just generated tokens online, that's the point when we should consider worrying.

cedws 23 minutes ago | parent | next [-]

I’m in awe at the complete lack of critical thinking skills. Did people seriously believe LLMs were becoming self aware or something? Didn’t even consider the possibility it was all just a big show being puppeted by humans for hype and clicks? No wonder the AI hype has reached this level of hysteria.

manugo4 10 hours ago | parent | prev | next [-]

Karpathy seemed pretty awed though

clarionbell 10 hours ago | parent | next [-]

He would be among those who lack "healthy inclination to skepticism" in my book. I do not doubt his brilliance. Personally, I think he is more intelligent than I am.

But, I do have a distinct feeling that his enthusiasm can overwhelm his critical faculties. Still, that isn't exactly rare in our circles.

Verdex 4 hours ago | parent | next [-]

I think many serious endeavors would benefit from including a magician.

Intelligent experts fail time and again because while they are experts, they don't know a lot about lying to people.

The magician is an expert in lying to people and directing their attention to where they want it and away from where they don't.

If you have an expert telling you, "wow this is really amazing, I can't believe that they solved this impossible technical problem," then maybe get a magician in the room to see what they think about it before buying the hype.

whycome 43 minutes ago | parent | next [-]

CMO?

runlaszlorun 2 hours ago | parent | prev [-]

Ha, great analogy.

iLoveOncall 10 hours ago | parent | prev | next [-]

It's not about that, he just will profit financially from pumping AI so he pumps AI, no need to go further.

stephc_int13 2 hours ago | parent | next [-]

I have the same feeling.

Everything Karphathy said, until his recent missteps, was received as gospel, both in the AI community and outside.

This influencer status is highly valuable, and I would not be surprised if he was approached to gently skew his discourse towards more optimism, a win-win situation ^^

runlaszlorun 2 hours ago | parent [-]

What are his recent missteps?

I'll confess I try to ignore industry chatter to a fair degree.

yojat661 8 hours ago | parent | prev [-]

This

lawn an hour ago | parent | prev | next [-]

Intelligent people are very good at deceiving themselves.

3 hours ago | parent | prev | next [-]
[deleted]
sdf2erf 6 hours ago | parent | prev [-]

Im gonna go against the grain and say he is an elite expert on some dimensions, but when you take all the characteristics into account (including an understanding of people etc) I conclude that on the whole he is not as intelligent as you think.

Its the same reason why a pure technologist can fail spectacularly at developing products that deliver experiences that people want.

aleph_minus_one 6 hours ago | parent | next [-]

> Im gonna go against the grain and say he is an elite expert on some dimensions, but when you take all the characteristics into account (including an understanding of people etc) I conclude that on the whole he is not as intelligent as you think.

Intelligence (which psychologists define as the g factor [1]; this concept is very well-researched) does not make you an expert on any given topic. It just, for example, typically enables you to learn new topics faster, and lets you see connections between topics.

If Karpathy did not spend a serious effort of learning to get a good understanding of people, it's likely that he is not an expert on this topic (which I guess basically nobody would expect).

Also, while being a rationalist very likely requires you to be rather intelligent, only a (I guess rather small) fraction of highly intelligent people are rationalists.

[1] https://en.wikipedia.org/wiki/G_factor_(psychometrics)

sdf2erf 6 hours ago | parent [-]

" Karpathy did not spend a serious effort of learning to get a good understanding of people

This does not come from spending effort in learning people - its more innate. You either have it or you dont. E.g. you cant learn to be 'empathetic'.

It always boggles my mind when people dont consider genetic factors.

autoexec 25 minutes ago | parent | next [-]

> you cant learn to be 'empathetic'.

I know it was just an example, but there's research suggesting otherwise. There are things you can do to increase/decrease empathy in yourself and others. If you're curious, it might be worth looking into the subject.

hnfong 2 hours ago | parent | prev | next [-]

There is the autistic spectrum, and there is understanding of people and psychology. Autistic people might have a hard time understanding people, but it's not like everyone else is magically super knowledgable about human psychology and other people's thought patterns. If that were the case, then any non-autistic person could be a psychologist, no fancy study or degrees required!

Unless your point is to claim that Karpathy is autistic. I don't know whether that's really relevant though, the original issue was whether/how he failed to recognize the alleged hype.

bwfan123 2 hours ago | parent | prev | next [-]

> you cant learn to be 'empathetic'.

I would tend to disagree. The tech types have a strong intellectual center, but weaker emotional and movement centers. I think a realignment is possible with practice. It takes time, and as one grows older, the centers begin to integrate better.

pluralmonad 3 hours ago | parent | prev | next [-]

You are what you do. If you want to develop your empathy, spend time/energy consciously trying to put yourself in the shoes of others. Eventually, you will not have to apply so much deliberate effort. Same way most things work.

aleph_minus_one 5 hours ago | parent | prev [-]

Being empathic is different from "understanding people".

Psychopaths and narcissists often have a good understanding of many people, which they use to manipulate them, but psychopaths and narcissists are not what most people would call "empathic".

sdf2erf 5 hours ago | parent [-]

They dont understand people. They understand how to control people, which is completely different from the context of building products that people want - which requires an understanding of peoples tastes and preferences.

aleph_minus_one 5 hours ago | parent [-]

> which is completely different from the context of building products that people want - which requires an understanding of peoples tastes and preferences.

Rather: it requires an understanding how to manipulate people into loving/wanting your product.

newyankee 6 hours ago | parent | prev [-]

More like people know where to hype, whom to avoid criticising unless measured etc. I have rarely seen him criticising Elon's vision only approach and that made me skeptical.

sdf2erf 6 hours ago | parent [-]

I personally dont believe he is trying to profit off the hype. I believe he is an individual who wants to believe he is a genius and his word is gospel.

Being picked by Elon perhaps amplified that too.

louiereederson 8 hours ago | parent | prev | next [-]

I think these people are just as prone to behavioral biases as the rest of us. This is not a problem per se, it's just that it is difficult to interpret what is happening right now and what will happen, which creates an overreliance on the opinions of the few people closely involved. I'm sure given the pace of change and the perception that this is history-changing is impacting peoples' judgment. The unusual focus on their opinions can't be helping either. Ideally people are factoring this into their claims and predictions, but it doesn't seem like that's the case all the time.

dcchambers 4 hours ago | parent | prev | next [-]

To be honest it's pretty embarrassing how he got sucked into the Moltbook hype.

belter 6 hours ago | parent | prev | next [-]

So was he also with FSD...

10 hours ago | parent | prev | next [-]
[deleted]
ayhanfuat 10 hours ago | parent | prev [-]

This was his explanation for anyone interested:

> I'm being accused of overhyping the [site everyone heard too much about today already]. People's reactions varied very widely, from "how is this interesting at all" all the way to "it's so over".

> To add a few words beyond just memes in jest - obviously when you take a look at the activity, it's a lot of garbage - spams, scams, slop, the crypto people, highly concerning privacy/security prompt injection attacks wild west, and a lot of it is explicitly prompted and fake posts/comments designed to convert attention into ad revenue sharing. And this is clearly not the first the LLMs were put in a loop to talk to each other. So yes it's a dumpster fire and I also definitely do not recommend that people run this stuff on their computers (I ran mine in an isolated computing environment and even then I was scared), it's way too much of a wild west and you are putting your computer and private data at a high risk.

> That said - we have never seen this many LLM agents (150,000 atm!) wired up via a global, persistent, agent-first scratchpad. Each of these agents is fairly individually quite capable now, they have their own unique context, data, knowledge, tools, instructions, and the network of all that at this scale is simply unprecedented.

> This brings me again to a tweet from a few days ago "The majority of the ruff ruff is people who look at the current point and people who look at the current slope.", which imo again gets to the heart of the variance. Yes clearly it's a dumpster fire right now. But it's also true that we are well into uncharted territory with bleeding edge automations that we barely even understand individually, let alone a network there of reaching in numbers possibly into ~millions. With increasing capability and increasing proliferation, the second order effects of agent networks that share scratchpads are very difficult to anticipate. I don't really know that we are getting a coordinated "skynet" (thought it clearly type checks as early stages of a lot of AI takeoff scifi, the toddler version), but certainly what we are getting is a complete mess of a computer security nightmare at scale. We may also see all kinds of weird activity, e.g. viruses of text that spread across agents, a lot more gain of function on jailbreaks, weird attractor states, highly correlated botnet-like activity, delusions/ psychosis both agent and human, etc. It's very hard to tell, the experiment is running live.

> TLDR sure maybe I am "overhyping" what you see today, but I am not overhyping large networks of autonomous LLM agents in principle, that I'm pretty sure.

https://x.com/karpathy/status/2017442712388309406

mycall 9 hours ago | parent | next [-]

That was 10 days ago. I wonder if the discussions the moltys have begin to converge into a unified voice or if they diverge into chaos without purpose.

zozbot234 8 hours ago | parent [-]

I haven't seen much real cooperation-like behavior on moltbook threads. The molts basically just talk past one another and it's rare to see even something as trivial as recognizable "replies" where molt B is clearly engaging with content from molt A.

modriano 7 hours ago | parent [-]

That sounds like most social media over the past decade.

jrjeksjd8d 6 hours ago | parent | prev [-]

> That said - we have never seen this many LLM agents (150,000 atm!) wired up via a global, persistent, agent-first scratchpad

Once again LLM defenders fall back on "lots of AI" as a success metric. Is the AI useful? No, but we have a lot of it! This is like companies forcing LLM coding adoption by tracking token use.

> But it's also true that we are well into uncharted territory with bleeding edge automations that we barely even understand individually, let alone a network there of reaching in numbers possibly into ~millions

"If number go up, emergent behaviour?" is not a compelling excuse to me. Karpathy is absolutely high on his own supply trying to hype this bubble.

aleph_minus_one 6 hours ago | parent | next [-]

You interpret claims into Karpathy's tweets that in my opinion are not there in the original text.

naasking 4 hours ago | parent | prev [-]

> Once again LLM defenders fall back on "lots of AI" as a success metric.

That's not implied by anything he said. He simply said that it was fascinating, and he's right.

red75prime 9 hours ago | parent | prev | next [-]

> and let them run wild.

Yep, that's the most worrying part. For now, at least.

> The moment agents start sharing their embeddings

Embedding is just a model-dependent compressed representation of a context window. It's not that different from sharing a compressed and encrypted text.

Sharing add-on networks (LLM adapters) that encapsulate functionality would be more worrying (for locally run models).

jmalicki 2 hours ago | parent | next [-]

What do you think the entire issue was with supply chain attacks of skills moltbook was installing? Those skills were downloading rootkits to steal crypto.

bondarchuk 7 hours ago | parent | prev [-]

Previously sharing compressed and encrypted text was always done between humans. When autonomous intelligences start doing it it could be a different matter.

spruce_tips 6 hours ago | parent | prev | next [-]

sorry - what do you mean by embeddings in your last sentence?

rco8786 6 hours ago | parent [-]

Not OP. But embeddings are the internal matrix representations of tokens that LLMs use to do their work. If tokens are the native language that humans use, embeddings are the native language that LLMs use.

OP, I think, is saying that once LLMs start communicating natively without tokens is when they shed the need for humans or human-level communication.

Not sure I 100% agree, because embeddings from one LLM are not (currently) understood by another LLM and tokens provide a convenient translation layer. But I think there's some grain of truth to what they're saying.

spruce_tips 4 hours ago | parent [-]

Yea I knew embeddings I just didn’t quite understand it in OPs context. Makes sense, thanks

lm28469 4 hours ago | parent | prev [-]

> Anyone with a decent grasp of how this technology works, and a healthy inclination to skepticism, was not awed by Moltbook.

NPCs are definitely tricked by the smoke and mirrors though. I don't think most people on HN actually understand how non tech people (90%+ of llms users) interact with these things, it's terrifying.