Absolutely. Like most things on the Internet, it will get enshittified. I think it is very likely that at some point there will be "ads" in the form of the chat bot giving recommendations that favor certain products and services.

  > Surely they know the risks, and surely people will be just as responsible with AI

What is the lesson in the anecdote about film students? To me, it’s that people like the idea of studying film more than they like actually studying film. I fail to see the connection to social media or AI.

> Film students who can no longer sit through films

Everyone loves watching films until they get a curriculum with 100 of them along with a massive reading list, essays, and exams coming up.

> surely people will be just as responsible with AI

That's exactly what worries us.

Perhaps the films were weren't worth sitting through?

Recently a side discussion came up - people in the Western world are "rediscovering" fermented, and pickled, foods that are still in heavy use in Asian cultures.

Fermentation was a great way to /preserve/ food, but it can be a bit hit and miss. Pickling can be outright dangerous if not done correctly - botulism is a constant risk.

When canning of foods came along it was a massive game changer, many foods became shelf stable for months or years.

Fermentation and pickling was dropped almost universally (in the West).

We lose something when we give up horses for cars.

Have too many of us outsourced our ability to raise horses for transport?

Surely you're capable of walking all day without break?

There's an Isaac Asimov story where people are "educated" by programming knowledge into their brains, Matrix style.

A certain group of people have something wrong with their brain where they can't be "educated" and are forced to learn by studying and such. The protagonist of the story is one of these people and feels ashamed at his disability and how everyone around him effortlessly knows things he has to struggle to learn.

He finds out (SPOILER) that he was actually selected for a "priesthood" of creative/problem solvers, because the education process gives knowledge without the ability to apply it creatively. It allows people to rapidly and easily be trained on some process but not the ability to reason it out.

I think the comparison to giving change is a good one, especially given how frequently the LLM hype crowd uses the fictitious "calculator in your pocket" story. I've been in the exact situation you've described, long before LLMs came out and cashiers have had calculators in front of them for longer than we've had smartphones.

I'll add another analogy. I tell people when I tip I "round off to the nearest dollar, move the decimal place (10%), and multiply by 2" (generating a tip that will be in the ballpark of 18%), and am always told "that's too complicated". It's a 3 step process where the hardest thing is multiplying a number by 2 (and usually a 2 digit number...). It's always struck me as odd that the response is that this is too complicated rather than a nice tip (pun intended) for figuring out how much to tip quickly and with essentially zero thinking. If any of those three steps appear difficult to you then your math skills are below that of elementary school.

I also see a problem with how we look at math and coding. I hear so often "abstraction is bad" yet, that is all coding (and math) is. It is fundamentally abstraction. The ability to abstract is what makes humans human. All creatures abstract, it is a necessary component of intelligence, but humans certainly have a unique capacity for it. Abstraction is no doubt hard, but when in life was anything worth doing easy? I think we unfortunately are willing to put significantly more effort into justifying our laziness than we will to be not lazy. My fear is that we will abdicate doing worthwhile things because they are hard. It's a thing people do every day. So many people love to outsource their thinking. Be it to a calculator, Google, "the algorithm", their favorite political pundit, religion, or anything else. Anything to abdicate responsibility. Anything to abdicate effort.

So I think AI is going to be no different from calculators, as you suggest. They can be great tools to help people do so much. But it will be far more commonly used to outsource thinking, even by many people considered intelligent. Skills atrophy. It's as simple as that.

I think you hit the nail on the head. Without years of learning by doing, experience in the saddle as you put it, who would be equipped to judge or edit the output of AI? And as knowledge workers with hands-on experience age out of the workforce, who will replace us?

The critical difference between AI and a tool like a calculator, to me, is that a calculator's output is accurate, deterministic and provably true. We don't usually need to worry that a calculator might be giving us the wrong result, or an inferior result. It simply gives us an objective fact. Whereas the output of LLMs can be subjectively considered good or bad - even when it is accurate.

So imagine teaching an architecture student to draw plans for a house, with a calculator that spit out incorrect values 20% of the time, or silently developed an opinion about the height of countertops. You'd not just have a structurally unsound plan, you'd also have a student who'd failed to learn anything useful.

That would have devastating consequences in the pre-LLM era, yes. What is less obvious is whether it'll be an advantage or disadvantage going forward. It is like observing that cars will make people fat and lazy and have devastating consequences on health outcomes - that is exactly what happened but the net impact was still positive because cars boost wealth, lifestyles and access to healthcare so much that the net impact is probably positive even if people get less exercise.

It is unclear that a human thinking about things is going to be an advantage in 10, 20 years. Might be, might not be. In 50 years people will probably be outraged if a human makes an important decision without deferring to an LLM's opinion. I'm quite excited that we seem to be building scaleable superintelligences that can patiently and empathetically explain why people are making stupid political choices and what policy prescriptions would actually get a good outcome based on reading all the available statistical and theoretical literature. Screw people primarily thinking for themselves on that topic, the public has no idea.

I believe that collectively we passed that point long before the onset of LLMs. I have a feeling that throughout the human history vast amounts of people ware happy to outsource their thinking and even pay to do so. We just used to call those arrangements religions.

I'll say that I'm still kinda on the fence here, but I will point out that your argument is exactly the same as the argument against calculators back in the 70s/80s, computers and the internet in the 90s, etc.

Too late. Outsourcing has already accomplished this.

No one is making cool shit for themselves. Everyone is held hostage ensuring Wall Street growth.

The "cross our fingers and hope for the best" position we find ourselves in politically is entirely due to labor capture.

The US benefited from a social network topology of small businesses. No single business being a lynch pin that would implode everything.

Now the economy is a handful of too big to fails eroding links between human nodes by capturing our agency.

I argued as hard as I could against shipping electronics manufacturing overseas so the next generation would learn real engineering skills. But 20 something me had no idea how far up the political tree the decision was made back then. I helped train a bunch of people's replacements before the telecom focused network hardware manufacturer I worked for then shut down.

American tech workers are now primarily cloud configurators and that's being automated away.

This is a decades long play on the part of aging leadership to ensure Americans feel their only choice is capitulate.

What are we going to do, start our own manufacturing business? Muricans are fish in a barrel.

And some pretty well connected people are hinting at similar sense of what's wrong: https://www.barchart.com/story/news/36862423/weve-done-our-c...

I'm grateful that I spent a significant part of my life forced to solve problems and forced to struggle to produce the right words. In hindsight I know that that's where all the learning was. If I'd had a shortcut machine when I was young I'd have used it all the time, learned much less, and grown up dependent on it.

Looking at the words that get produced at this lowered cost, and observing how satisfactory they apparently are to most people (and observing the simplicity of the heuristics people use to try to root out "cheap" words), has been quite instructive (and depressing).

Yet it does feel different with LLMs compared to your examples. Yes, people can’t navigate without Apple/Google maps, but that’s still very different from losing critical thinking skills.

That said, LLMs are perhaps accelerating that but aren’t the only cause (lack of reading, more short form content, etc)

> it’s hard to get back once the habit forms.

Humans are highly adaptable. It's hard to go back while the thing we're used to still exists, but if it vanished from the world we'd adapt within a few weeks.

The main difference is that the computer you use for writing is not requiring you to pay for every word. And that's the difference in the business models being pushed right now all around the world.

If an AI thinks for you, you're no longer "outsourcing" parts of your mind. What we call "AI" now is technically impressive but is not the end point for where AI is likely to end up. For example, imagine an AI that is smart enough to emotionally manipulate you, at what point in this interaction do you lose your agency to "outsource" yourself instead of acting as a conduit to "outsource" the thoughts of an artificial entity? It speaks to our collective hubris that we seek to create an intellectually superior entity and yet still think we'll maintain control over it instead of the other way around.

I usually feed my articles to it and ask for insight into whats working. I usually wait to initiate any sort of AI insight until my rough draft is totally done...

Working in this manner, it is so painfully clear it doesnt really follow the flow of the article even. It misses on so many critical details and just sorta fills in its own blanks wrong... When you tell it that its missing a critical detail, it treats you like some genius, every single time.

It is hard for me to try to imagine growing up with it, and using it to write my own words for me. The only time i copy paste words to a fellow human that is ai generated, is for totally generic customer service style replies, for questions i dont totally consider worthy of any real time.

AI has kinda taken away my flow state for coding, rare as it was... I still get it when writing stuff I am passionate about, and I can't imagine I'll ever wanna outsource that.

> And it terrifies me that others aren’t quite as objective.

I have been reminded constantly throughout this that a very large fraction of people are easily impressed by such prose. Skill at detecting AI output (in any given endeavour), I think, correlates with skill at valuing the same kind of work generally.

Put more bluntly: slop is slop, and it has been with us for far longer than AI.

Executive function is not the same as weaving or carpentry. The scary problem comes from people who are trying to abdicate their entire understand-and-decide phase to an outside entity.

What's more, that's not fundamentally a new thing, it's always been possible for someone to helplessly cling to another human as their brain... but we've typically considered that to be a mental-disorder and/or abuse.

> How many of you know how to [...] cook your own food?

That's a very low bar. I expect most people know how to cook, at least simple dishes.

Systems used to be robust, now they’re fragile due to extreme outsourcing and specialization. I challenge the belief that we’re getting along fine. I argue systems are headed to failure, because of over optimization that prioritized output over resilience.

What is that saying again, a person is smart, a group is dumb?

That's the risk involved with opinions and conclusions.

Wasn't there a follow-up to this where Scott denied that the story was "about" the obvious thing for it to be about?

> Social media has given me a rather dim view of the quality of people's thinking, long before AI. Outsourcing it could well be an improvement.

Cogito, ergo sum

The corollary is: absence of thinking equals non-existence. I don't see how that can be an improvement. Improvement can happen only when it's applied to the quality of people's thinking.

> If outsourcing thought is beneficial, those who practice it will thrive

It makes them prey to and dependent on those who are building and selling them the thinking.

> I suspect that the best results would be achieved by outsourcing relatively small intellectual acts in a way that guarantees very rare, very small errors. That is, AI will become useful when AI becomes dependable, comparable to our other tools.

That's like saying ultra processed foods provide the best results when eaten sparingly, so it will become useful when people adopt overall responsible diets. Okay, sure, but what does that matter in practice since it isn't happening?

Outsourcing thinking is not a skill. It is the same as skipping gym. Nothing to practice here