I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability.

Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/

> At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were not interesting to him, because they were about building up a solid AI foundation, then the AI system runs as a sort of black box. "I'm not interested in that. I want software that's accountable." Accountable? "Yes, I want something that can express its symbolic reasoning. I want to it to tell me why it did the thing it did, what it thought was going to happen, and then what happened instead." He then said something that took me a long time to process, and at first I mistook for being very science-fiction'y, along the lines of, "If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court."

Years later, I found out that Sussman's student Leilani Gilpin wrote a dissertation which explored exactly this topic. Her dissertation, "Anomaly Detection Through Explanations", explores a neural network talking to a propagator model to build a system that explains behavior. https://people.ucsc.edu/~lgilpin/publication/dissertation/

There has been followup work in this direction, but more important than the particular direction of computation to me in this comment is that we recognize that it is perfectly reasonable to hold AI corporations to account. After all, they are making many assertions about systems that otherwise cannot be held accountable, so the best thing we can do in their stead is hold them accountable.

But a much better path would be to not use systems which fail to have these properties, and expand work on systems which do.

My team and I are firm that we are the ones accountable. LLMs are a tool like every other. Only that it's non deterministic. But I am the one using the tool. I am the one giving the tool access. I am the one who has to keep everything safe.

I have shot myself in the foot using gparted in the past by wiping the wrong disk. gparted wasn't to blame. I was.

Letting LLMs work freely without supervision sounds great but it will lead to pain. I have to supervise their work. And that is also during execution. You can try to replace a human but we see where this leads. Sooner or later the LLM will do something stupid and then the only one to blame is the person who used the tool.

When I was a masters student in STS[1], one of my concepts for a thesis was arguing that one of the primary uses of software was to shift or eschew agency and risk. Basically the reverse of the famous IBM "a computer can not be held responsible" slide. Instead, now companies prefer computers be responsible because when they do illegal things they tend to be in a better legal position. If you want to build as tool that will break a law, contract it out and get insurance. Hire a human to "supervise" the tool in a way they will never manage and then fire them when they "fail." Slice up responsibility using novel command and control software such that you have people who work for you who bear all the risk of the work and capture basically none of the upside.

It's not just AI. It's so much of modern software - often working together with modern financialization trends.

[1] Basically technology-focused sociology for my purposes, the field is quite broad.

Accountability is the prevailing missing ingredient in us society.

I think there is a much more fundamental question about "tooling."

Quoth the author: "But I also know you can't blame a tool for your own mistakes."

Are we able to completely classify any and all AI models as tools? Or are they something more?

I don't know the answer to this question.

People are eschewing their own accountability, blaming the tools instead for their poor decision making and lack of access controls.

Why is it possible for you to fat-finger your way to deleting production database locally?

Have I got a book for you: https://en.wikipedia.org/wiki/The_Unaccountability_Machine

Not actually about technology at all, but about organizational structure.

Another view of the accountability is that we're currently often pointing accountability in the wrong direction, and it's gaining momentum. Aspects of it have been around so long it's a trope: important work around maintainability is undervalued.

Imagine two parallel universes:

- in one, you take ten minutes to make a dashboard that shows management what they asked for. It passes code review before merge and the exec who asked for it says it's what they wanted.

- in the other, you take a day or two to make it. Again, it passes code review before merge and the exec who asked for it says it's what they wanted.

Which version of you is more likely to get positive versus negative feedback? Even if the quick-to-build version isn't actually correct? If you're too slow and aren't doing enough that looks correct, you'll be held accountable. But if you're fast and do things that look correct but aren't, you won't be held accountable. You'll only be held accountable for incorrect work if the incorrectness is observed, which is rarer and rarer with fewer and fewer people directly observing anything.

So oddly, with nobody doing it on purpose, people get held accountable specifically for building things the way you're advocating.

I imagine that orgs that do lots of incorrect work could be outcompeted but won't be, because observability is hard and the "not get in trouble" move is to just not look too hard at what you're doing and move to the next ticket.

If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court.

How would that work? You have the AI explain its reasoning - and trust that this is accurate - and then you decide whether that is acceptable behavior. If not, you ban the AI from driving because it will deterministically or at least statistically repeat the same behavior in similar scenarios? Fine, I guess, that will at least prevent additional harm. But is this really all that you want? The AI - at least as we have them today - did not create itself and choose any of its behaviors, the developers did that. Would you not want to hold them responsible if they did not properly test the AI before releasing it, if they cut corners during development? In the same way you might hold parents responsible for the action of their children in certain circumstances?

> The problem is that people are now building our world around tooling that eschews accountability.

If by "now" you mean "for the past few decades", I think you've got it spot on, at least per the very interesting https://en.wikipedia.org/wiki/The_Unaccountability_Machine

The key quote is in the increasingly prescient 1979 IBM training manual: "A computer can never be held accountable, therefore a computer must never make a management decision."

That manual aged much more gracfully than the 1930s "Songs of the IBM," featuring lines like "The name of T.J. Watson means a courage none can stem / And we feel honored to be here to toast the I.B.M.," and of course classic American standards like "To G.H. Armstrong, Sales Manager, ITR and IS Divisions."

There used to be a lot of research into using deep NNs to train decision trees, which are themselves much less of a black box and can actually be reasoned about. I wonder where that all went?

The fallacy here is the assumption that humans know why we do what we do. Much like modern LLMs we have an explanation, but it’s just something we cook up in our brain. Whether or not it’s the truth is far more complex.

Oddly, despite LLMs being these huge networks with billions of parameters, we still probably do understand it better than we do our own brains.

About the blog you linked and not your comment:

Doesn't symbolic AI have a lot of philosophical problems? Think back to Quine's two dogmas - you can't just say, "Let's understand the true meanings of these words and understand the proper mappings". There is no such thing as fixed meaning. I don't see how you get around that.

Deep learning is admittedly an ugly solution, but it works better than symbolic AI at least.

That is part of why https://mieza.ai/ is giving a grounding layer that is backed by game theory. Actions have consequences. Tracking decisions and their consequences is important.

One thing that becomes very clear from this sort of work is just how bad LLMs are. It can be invisible when you're working with them day to day, because you tend to steer them to where they are helpful. Part of game theory though is being robust. That means finding where things are bad, too, not just exploring happy paths.

To get across just how bad the failure cases of LLMs are relative to humans, I'll give the example of tic tac toe. Toddlers can play this game perfectly. LLMs though, don't merely do worse than toddlers. It is worse then that. They can lose to opponents that move randomly.

They can be just as bad as you move to more complex games. For example, they're horrible at poker. Much worse than human. Yet when you read their output, on the surface layer, it looks as if they are thinking about poker reasonably. So much so, in fact, that I've seen research efforts that were very misguided: people trying to use LLMs to understand things about bluffing and deception, despite the fact that the LLMs didn't have a good underlying model of these dynamics.

It is hard to talk about, because there are a lot of people who were stupid in the past. I remember people saying that LLMs wouldn't be able to be used for search use-cases years back and it was such a cringe take then and still is that I find myself hesitant to talk about the flaws. Yet they are there. The frontier is quite jagged. Especially if you are expecting it to be smooth, expecting something like anything close to actual competence, those jagged edges can be cutting and painful.

Its also only partially solvable through scale. Some domains have a property where, as you understand it better, the options are eliminated and constrained such that you can better think about it. Game theory, in order to reduce exploitability, explores the whole space. It defies minimization of scope. That is a problem, since we can prove that for many game theoretic contexts, the number of atoms is eclipsed by the number of unique decisions. Even if we made the model the size of our universe there would still be problems it could, in theory, be bad at.

In short, there is a practical difference between intelligence and decision management, in much the same way there is a practical difference between making purchases and accounting. And the world in which decisions are treated as seriously as they could be so much so exceeds our faculties that most people cannot even being to comprehend the complexity.

It's taking "computer says no" to the next level. Computers do exactly what they're told, but who told them? The person entering data? The original programmer or designer of the system? The author of whatever language text was used to feed the ai? Even before AI, it was very difficult to determine who is accountable, and now it's even more obfuscated.

> The problem is that people are now building our world around tooling that eschews accountability.

If you tell Terraform the wrong thing it will remove your database and not be accountable either.

> The problem is that people are now building our world around tooling that eschews accountability.

Tools cannot eschew accountability. But the users of the tools can and that is exactly what happened in the PocketOS fiasco.

Just as a company is responsible for the actions of its junior employees, so too are users responsible for their LLMs.

"It is a poor workman who blames his tools."

Humans aren’t any better. That’s why we have OSHA etc. I think you’re hoping for a formal logic based AI and I’ll wager no such thing will ever exist - and if it do, it would try to kill us all.

leilani's work is super interesting

I feel like "AI didn't delete your database, you did" is all about who has accountability, though.

Very informative post. I think however we are not at the point AI can be taken to court. We know it can hallucinate, we know that context can fill up or obfuscate a rule and cause behaviour we explicitly didn't want.

If you give the AI agency to execute some task, you are still responsible. In the near term we should focus on tooling for auditing and sandboxing, and human in the loop confirmations.

> so the best thing we can do in their stead is hold them accountable

We can't even do this. They are worth too much money already to ever be held really accountable.

The best we can ever hope for is they might occasionally be hit with relatively insignificant "cost of doing business" fines from time to time.

So this starts out very interesting then the “symbolic reasoning” cult stuff kicks in.

Why is there a group of people always obsessed with symbolic reasoning being the only way AI can function and regularly annoy explain why humans (who are not strict symbolic reasoning machines at any level) work.

[dead]

I don’t know why people still consider the US the ideal country for starting companies. Everything seems to evolve around taking people to court.

I wish you could have what you want, but I worry you won't get this, because life doesn't give you that, and these systems are tending away from machine precision, and more toward life-like trade-offs.

I am almost certain that even if you did get what you want, something that isn't what you want will run circles around you and eat your lunch

EDIT: I suspect this will be an unpopular take on Hacker News. And so I am soliciting upvotes for visibility from other biologists and sympathetic technologists. I think everyone should try to grapple with this possibility <3