Overall, I love this essay. However, the entire argument hinges on one assertion, buried about halfway through:

> Robots are improving fast, but I do not believe that this cute fellow will be stuffing envelopes or affixing stamps anytime soon.

Is this correct? I don't feel qualified to say. But if it's wrong... well, then there's a missing pixel in the magic circle, and flood fill will make the whole thing unrecognizable.

His (compelling) evidence for that assertion is that printers still jam after 40 years. For humans, writing something on a piece of paper is absolutely trivial, and if something goes wrong, grabbing a new piece of paper or a pen is also trivial. Computers _can_ now write on paper tolerably fast and well, but they absolutely can't handle even simple failure modes. And the real world is _massively_ failure-prone, in contrast to the digital domain.

Think about Tesla's pivot to "AI robots". My guess is that they'll get to something that can very slowly pick up a dropped sock and put it in the washing basket. But that it will fall over occasionally on the stairs, wrecking your kid's photos and the vase standing at the bottom, and dinging the wall. It might do a passable job of picking up the shards of pottery, but gluing the picture frames together, plastering the wall and repainting it... well maybe in in Elon's chemical dreams.

I also love this essay, but I think there's a much larger, scarier breach in the magic circle.

We humans consume information on the Internet, it changes our ideas, and those ideas directly inform our very physical and material behavior. We ourselves are essentially 3D printers for our thoughts, running 24/7.

Flashmobs, scenic spots that get overrun with tourists after an Instagram post goes viral, teens eating tide pods, adults failing to cure COVID with Ivermectin, fashion trends, everyone kind of getting into sourdough during the pandemic, Kate Bush making almost half a million bucks in two weeks because of Stranger Things, the death of Payton Isabella Leutner, millions of people protesting for Black Lives Matter, and thousands more are real-world events that would not have happened without the Internet infecting brains.

Elections are decided based on what people learn online, and those elections have world-sized potentially catastrophic impact when you consider things like climate change policy.

I fear there is no meaningful separation between the digital world and the physical world, because it's really about the separation between ideas and material reality. Living beings exist entirely to span that bridge.

Like Rodney Brooks says, "No one has managed to get articulated fingers (i.e., fingers with joints in them) that are robust enough, have enough force, nor enough lifetime, for real industrial applications."

Here, I'll link to that piece directly, it's long and detailed and illustrated, and it also counters the idea of just throwing AI at the problem until robot dexterity emerges from whatever physical parts.

https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex...

"there have now been fifteen different families of neurons discovered that are involved in touch sensing and that are found in the human hand" ... "a human hand has about 17,000 low-threshold mechanoreceptors" ... "These receptors come in four varieties (slow vs fast adapting, and a very localized area of sensitivity vs a much larger area)"

You might ask, do robots that interact with the real world need such complicated bio-mimicking physical tech, or can they cut corners? But they can't cut all the corners, anyway. Somebody has to make a high-bandwidth robot hand with flexible strength and a self-repair ability. Or, hey, cyborgs maybe? Reanimate cadavers with AI, that could do the trick.

The sensing part is really really hard.

Hands, mechanically are fairly simple to mimic, touch? the way skin feels micron-level shifts and subtle temperature changes is plain tough.

Even if the assertion is correct, what would life be like with computers incomprehensibly smarter and faster thinking than humans?

All management decisions from the top down to individual manual workers handled by an AI (LLM or otherwise)?

Owner has a company-wide AI, instructs it to maximize profit and lets it run. It handles hiring and firing, marker research, advertisement, ordering supplies, ... It generates individualized instructions for each worker what to do throughout the day. Any communication between humans would be redundant, the AI would have microphones and cameras everywhere, humans would only be needed for physical interaction with the world. Even communication with other companies, suppliers and clients, would by done between AIs, they would be better and faster at negotiating.

1) It sounds like a dystopian nightmare - constant surveillance and taking orders from a machine which only cares your productivity.

2) Would it lead to a devolution of the human race? What makes us different from other animals is intelligence. If all humans are good at (= economical to use for) is manual labor, would intelligence stop being a sexually attractive trait?

3) It would completely remove any social mobility. If those who own companies continue owning them after an AI revolution and there's little economic value in human labor except the most menial, then there would be a 2 class society with almost no way for non-owners to become owners.