I've made that point before. From an "alignment" perspective, AIs and corporations have many of the same problems. Corporations, according to Milton Friedman, have the sole duty of increasing their profits.[1] This is a key document in the history of modern conservatism.[2]

So what happens when AIs are "aligned" purely to corporate goals? This is a nearer term worry than machines taking over for the benefit of machines. AIs working hard for the benefit of corporations are already here. Try any customer service chatbot.

There's reluctance to think about this too hard. It raises embarrassing questions about the role of the corporation in society. It can become a challenge to corporate personhood. That's threatening to many people in power. Dealing with it may be unavoidable as AIs take a stronger role in managing companies.

[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...

[2] https://www.heritage.org/trade/report/milton-friedman-the-fa...

Very true!

Also the accountability problem! Sometimes we try to hold the corporation accountable and sometimes the individuals. It never seems adequate though.

I found the documentary "The Corporation" (2003) very instructive in that regard, particularly the segment with the BP CEO where the point is made that a company made up ethical individuals can still act unethically, or unaligned to human values, just by virtue of the incentive and legal structure governing the corporation.

Thanks for the reply. I was hoping to get some debate going and have my thoughts challenged in order to see how they hold up under scrutiny. You're right that I believe in Functionalism and Strong AI and in previous comment I only really stated my belief so please allow me to try and make a bit of an argument for it. My position probably falls under the "System reply" to the Chinese Room argument although I feel it's more restrictive in some aspects so perhaps it corresponds to a particular subtype (I don't know as I haven't read enough in the area).

My basic argument is that human *intelligence* (and understanding - I consider understanding as part of intelligence and consciousness separately but I can see how many might group understanding with consciousness and leave intelligence on its own) is a form of collective intelligence. Individual neurons acting based on local chemical and electrical signals collectively produce behaviour that we call intelligent, i.e. it can solve problems, respond to questions, etc ... . A particular configuration of neurons acting together I will call a particular software or algorithm (I understand that separating the software from the neural substrate in that way is part of what's in dispute, particularly by Searle, but that's my belief and what I feel is required to make the mapping to the Chinese Room argument). The Chinese Room thought experiment then is trying to refute Strong AI by arguing that the person in the Chinese Room has no understanding of the exchange with the outside world because they are just manipulating symbols incomprehensible to them. I agree that the person has no understanding. That person on their own would not be able to string together a sensible reply without the instruction cards. However another person with the same instruction cards, i.e. algorithm, would be able to reply sensibly, pass a Chinese Turing test, etc ..., thus demonstrating that the intelligence/understanding resides in the algorithm. The person is just an execution substrate. The mistake in the Chinese Room argument is identifying the locus of understanding in the person, which imho is failure of imagination. It's akin to saying that humans can have no understanding because each individual neuron has no understanding of our actions, something which the existence proof of human intelligence refutes.

What then of consciousness? Here I also have some beliefs which I'm happy to write down but in my assessment they're less substantiated so make of them what you will. I saw some things by Joscha Bach recently and I liked what he said about agents and consciousness as a form self-recognition of the agent as an entity in the environment and predictive modeling of future states of that entity in order to achieve the agents goals and realise some form of agency. I don't know where neuroscience and cognitive science stand on this but made sense to me as a sensible model of consciousness. Therefore I would even go so far as to say that I believe that entities such as the United States of America (and any country or institution really) are probably conscious in some form, because they are aware of themselves, model themselves, and try to project their agency in the environment that they find themselves in. We are not directly aware of this because we are like single neurons in our brain and this happens at a scale much larger than us. Moreover I suspect that as you go up each rung on the emergence ladder, the temporal scale of the phenomena gets an order slower as that level is composed of the interactions at the level below it. So the speed of interactions of our neurons is an order faster than our conscious processes and similarly the emergent bodies and institutions that we form part of act an order slower so their processes are more difficult for us to perceive.

Schematically then my beliefs organise roughly as follows:

- intelligence = algorithm: think neural network weights you can download of HuggingFace

- consciousness = self-aware world model + state: I speculate that the same NN world model weights running with separate state would perceive themselves as separate entities

Would love to hear some thoughts on this!

Agreed. Michael Levin's work and discoveries are mind blowing! I was actually referencing that with the "electrical signals" nod (as well as neuronal activity) and Alan Turing's reaction-diffusion theory of morphogenesis with the chemical gradients. I'm not a molecular biologist so i don't know anything about the details, but the emergence of global phenomena from these local interactions fascinates me.

There's no intelligence without teleology. Intelligence and information are only meaningful concepts of you assert (necessarily without any evidence except the circular kind) that some differences matter.

Do the cells "have" to do anything? Do we? Sure, they might die if they do the "wrong" thing, but 1. Who's to say the inert state of "dead" matter is any worse than the state of a living cell, and 2. Cells clearly aren't always optimizing for their own survival anyway

> But why would you do that?

Because that is what actually gets pursued. The set of effective goals an organization is distinctly different from those of any of the individuals in that organization.

I kinda agree that the financial system is intelligent in the problem solving sense but disagree that it is pursuing a particular goal in the sense that institutions and organisations are.

It's more akin to an ecosystem to me. In nature we also have ecosystems organising and reusing resources but I would only ascribe intelligence to living organisms pursuing individual goals with agency.

Haven't thought too hard about the distinction though so happy to be shown a better principle.

> Markets solve the resource allocation problem

Not really. If they did, markets would be more stable. Look at the prices of commodities such as copper. Prices swing up and own by as much as 3x, despite both supply and demand not changing much. That's the problem. If supply and demand have a lot of inertia, and thus lag, the feedback loop becomes unstable. This is basic control theory. Hedging with derivatives was supposed to add stability, but doesn't work very well for that purpose.

I agree with you on the markets but not on institutions and organisations.

Organisations pursue goals with agency and there is even Darwinian competition between them in many cases.

> Markets solves the resource allocation problem

Then why do we still have homeless and jobless?

Cool, will check it out.

Yeah, wasn't claiming that it was particularly novel or divine insight, otherwise I might have tried to publish a paper about it. I was just recounting my experience on this matter.

They're de facto superhuman. Even with the friction and stupidity, they still have more agency than an individual human and process a much bigger entropic footprint for both information and physical resources.

You occasionally get Great Leaders who build these systems as personal affordances. But even then these system are far beyond any one person's ability to manage the details. And they often take on a life of their own.

Our individuality is actually a myth. We're more like a colony organism that self-organises into different sub-organisms. The sub-organisms compete with each other, which is why we're not a smart colony organism.

If individual intelligence was aligned along colony goals at shared planetary scale we'd be something else entirely.

The biological systems that support human intelligence are notoriously messy and work together only tenuously with a great many failure modes, including "process for the sake of process"