It's not as black-and-white as "Brenda good, AI bad". It's much more nuanced than this.

When it comes to (traditional) coding, for the most part, when I program a function to do X, every single time I run that function from now until the heat death of the sun, it will always produce Y. Forever! When it does, we understand why, and when it doesn't, we also can understand why it didn't!

When I use AI to perform X, every single time I run that AI from now until the heat death of the sun it will maybe produce Y. Forever! When it does, we don't understand why, and when it doesn't, we also don't understand why!

We know that Brenda might screw up sometimes but she doesn't run at the speed of light, isn't able to produce a thousand lines of Excel Macro in 3 seconds, doesn't hallucinate (well, let's hope she doesn't), can follow instructions etc. If she does make a mistake, we can find it, fix it, ask her what happened etc. before the damage is too great.

In short: when AI does anything at all, we only have, at best, a rough approximation of why it did it. With Brenda, it only takes a couple of questions to figure it out!

Before anyone says I'm against AI, I love it and am neck-deep in it all day when programming (not vibe-coding!) so I have a full understanding of what I'm getting myself into but I also know its limitations!

No contradiction here:

When we say “machine”, we mean deterministic algorithms and predictable mechanisms.

Generative AI is neither of those things (in theory it is deterministic but not for any practical applications).

If we order by predictability:

Quick Sort > Brenda > Gen AI

"Thinking mode" only provides the illusion of debuggability. It improves performance by generating more tokens which hopefully steer the context towards one more likely to produce the desired response, but the tokens it generates do not reflect any sort of internal state or "reasoning chain" as we understand it in human cognition. They are still just stochastic spew. You have no more insight into why the model generates the particular "reasoning steps" it does than you do into any other output, and neither do you have insight into why the reasoning steps lead to whatever conclusion it comes to. The model is much less constrained by the "reasoning" than we would intuit for a human - it's entirely capable of generating an elaborate and plausible reasoning chain which it then completely ignores in favor of some invisible built-in bias.

> We disavow AI because people like Brenda are perfect and the machine is error-prone.

No, no. We disavow AI because our great leaders inexplicably trust it more than Brenda.

There are other narratives going on in the background though both called out by the article and implied, including:

Brenda probably has annual refresher courses on GAAP, while her exec and the AI don't.

Automation is expected to be deterministic. The outputs can be validated for a given input. If you need some automation more than Excel functions, writing a power automate flow or recording an office script is sufficient & reliable as automation while being cheaper than AI. Can you validate AI as deterministic? This is important for accounting. Maybe you want some thinking around how to optimize a business process, but not for following them.

Brenda as the human-in-the-loop using AI will be much more able than her exec. Will Brenda + AI be better (or more valuable considering the cost of AI) than Brenda alone? That's the real question, I suppose.

AI in many aspects of our life is simply not good right now. For a lot of applications, AI is perpetually just a few years away from being as useful as you describe. If we get there, great.

The promise of AI is that it lets you "skip the drudgery of thinking about the details" but sometimes that is exactly what you don't want. You want one or more humans with experience in the business domain to demonstrate they have thought about the details very carefully. The spreadsheet computes a result but its higher purpose is a kind of "proof" this thinking was done.

If the actual thinking doesn't matter and you just need some plausible numbers that look the part (also a common situation), gen ai will do that pretty well.

> So, then - why don't people embrace AI with thinking mode as an acceptable form of automation?

"Thinking" mode is not thinking, it's generating additional text that looks like someone talking to themselves. It is as devoid of intention and prone to hallucinations as the rest of LLM's output.

> Can't the C-suite in this case follow its thought process and step in when it messes up?

That sounds like manual work you'd want to delegate, not automation.

Brenda has years (hopefully) of institutional knowledge and transferrable skills.

"hmm, those sales don't look right, that profit margin is unusually high for November"

"Last time I used vlookup I forgot to sort the column first"

"Wait, Bob left the company last month, how can he still be filing expenses"

The “Brenda” example is a lumped sum fallacy where there is an “average” person or phenomenon that we can benchmark against. Such a person doesn't exist, leading to these dissonant, contradictory dichotomies.

The fact of the matter is that there are some people who can hold lots of information in their head at once. Others are good at finding information. Others still are proficient at getting people to help them. Etc. Any of these people could be tasked with solving the same problem and they would leverage their actual, particular strengths rather than some nebulous “is good or bad at the task” metric.

As it happens, nearly all the discourse uses this lumped sum fallacy, leading to people simultaneously talking past one another while not fundamentally moving the discussion forward.

That automation you cite in your #1 is advocated for because it is deterministic and, with effort, fairly well understood (I have countless scripts solidly running for years).

I don't disavow AI, but like the author, I am not thrilled that the masses of excel users suddenly have access to Copilot (gpt4). I've used Copilot enough now to know that there will be huge, costly mistakes.

> We disavow AI because people like Brenda are perfect and the machine is error-prone.

I don't think that is the message here. The message is that while Brenda might know what she is doing and maybe AI helps her.

> She's gonna birth that formula for a financial report and then she's gonna send that financial report

The problem is people who might not know what they are doing

> he would have sent it back to Brenda but he's like oh I have AI and AI is probably like smarter than Brenda and then the AI is gonna fuck it up real bad

Because AI outputs sound so confident it makes even the layman feel like an expert. Rather than involve Brenda to debug the issue, C-suite might say - I believe! I can do it too. AI FTW!

Even when people advocate automation especially in areas like finance there is always a human in the loop whose job is to double check the automation. The day when this human finds errors in the machine there is going to be lot of noise. And if the day happens to be a quarterly or yearly closing/reporting there is going to be hell to pay once closing/reporting is done. Both the automation and developer are going to be hauled up (obviously I am exaggerating here).

The issue is reliability.

would you be willing to guarantee that some automation process will never mess up, and if/when it does, compensate the user with cash.

For a compiler, with a given set of test suites, the answer is generally yes, and you could probably find someone willing to insure you for a significant amount of money, that a compilation bug will not screw up in a such a large way that it will affect your business.

For a LLM, I have a believing that anyone will be willing to provide that same level of insurance.

If a LLM company said "hey use our product, it works 100% of the time, and if it does fuck up, we will pay up to a million dollars in losses" I bet a lot of people would be willing to use it. I do not believe any sane company will make that guarantee at this point, outside of extremely narrow cases with lots of guardrails.

That's why a lot of ai tools are consumer/dev tools, because if they fuck up, (which they will) the losses are minimal.

> So, then - why don't people embrace AI with thinking mode as an acceptable form of automation

Mainly because Generative AI _is not automation_ . Automation is set on fixed ruleset, predictable, reliable and actually saving time. Generative AI ...is whatever it is, it is definitely not automation.

I feel like it comes down to predictability and overall trust and confidence. AI is still very fucky, and for people that don't understand the nuances, it definitely will hallucinate and potentially cause real issues. It is about as happy as a Linux rm command to nuke hours of work. Fortunately these tools typically have a change log you can undo, but still.

Also Brenda is human and we should prioritize keeping humans in jobs, but with the way shit is going that seems like a lost hope. It's already over.

Humans, legacy algorithmic systems, and LLM's have different error modes.

- Legacy systems typically have error modes where integrations or user interface breaks in annoying but obvious ways. Pure algorithms calculating things like payroll tend to be (relatively) rigorously developed and are highly deterministic.

- LLMs have error modes more similar to humans than legacy systems, but more limited. They're non-deterministic, make up answers sometimes, and almost never admit they can't do something; sometimes they make pure errors in arithmetic or logic too.

- Humans have even more unpredictable error modes; on top of the errors encountered in LLM's, they also have emotion, fatigue, org politics, demotivation, misaligned incentives, and so on. But because we've been dealing with working with other humans for ten thousand years we've gotten fairly good at managing each other... but it's still challenging.

LLMs probably need a mixture of "correctness tests" (like evals/unit tests) and "management" (human-in-the-loop).

In my opinion there's a big difference in deterministic and nondeterministic automation.

By the same fascination, do computers become more complex to enhance people? or do people get more complex with the use of computers? Also, do computers allow people to become less skilled and inefficient? or do less skilled and inefficient people require the need for computers?

The vector of change is acceptable in one direction and disliked in another. People become greater versions of themselves with new tech. But people also get dumber and less involved because of new tech.

This misunderstands complexity entirely:

The complexity of the task isn't a factor - it's complex to generate correct machine code, but we trust compilers to do it all the time.

The big problem with AI in back-office automation is that it will randomly decide to do something different than it had been doing. Meaning that it could be happily crunching numbers accurately in your development and launch experience, then utterly drop the ball after a month in production.

While humans have the same risk factors, human oriented back-office processes involve multiple rounds of automated/manual checks which are extremely laborious. Human errors in spreadsheets have particular flavors such as forgotten cell, misstyped number, or reading from the wrong file/column. Human's are pretty good at catching these errors as they produce either completely wrong results when the columns don't line up - or the typo'd number is completely out of distribution.

An AI may simply decide to hallucinate realistic column values rather than extracting its assigned input. Or hallucinate a fraction of column values. How do you QA this? You can't guarantee that two invocations of the AI won't hallucinate the same values, you can't guarantee that a different LLM won't hallucinate different values. To get a real human check, you'd need to re-do the task as a human. In theory you can have the LLM perform some symbolic manipulation to improve accuracy... but it can still hallucinate the reasoning traces etc.

If a human decided to make up accounting numbers one out of every 10000 accounting requests they would likely be charged with fraud. Good luck finding the AI hallucinations at the equivalent level before some disaster occurs. Likewise, how do you ensure the human excel operator doesn't get pressured into certifying the AIs numbers when the "don't get fired this week" button is sitting right their in their excel app? how do you avoid the race to the bottom where the "star" employee is the one certifying the AI results without thorough review?

I'm bullish on AI in backoffice, but ignoring the real difficulties in deployment doesn't help us get there.

I feel like you've squashed a 3D concern (automations at different levels of the tech stack) into a 2D observation (global concerns about automations).

Human determinism, as elastic as it might be, is still different than AI non-determinism. Especially when it comes to numbers/data.

AI might be helpful with information but it's far less trustable for data.

It is of course because algorithms can be repaired when they are buggy, but a large language model can not, because it is impossible to look at its weights and say, look, this is where the mistakes has happened.

The reason is oftentimes fairly simple, certain people have their material wealth and income threatened by such automation, and therefore it's bad (an intellectualized reason is created post-hoc)

I predict there will actually be a lot of work to be done on the "software engineering" side w.r.t. improving reliability and safety as you allude to, for handing off to less than sentient bots. Improved snapshot, commit, undo, quorum, functionalities, this sort of thing.

The idea that the AI should step into our programs without changing the programs whatsoever around the AI is a horseless carriage.

Non deterministic vs deterministic automation

> it's complex to generate correct machine code, but we trust compilers to do it all the time.

Generating correct machine code is actually pretty simple. It gets complicated if you want efficient machine code.

> So, then - why don't people embrace AI with thinking mode as an acceptable form of automation? Can't the C-suite in this case follow its thought process and step in when it messes up?

> I think people still find AI repugnant in that case. There's still a sense of "I don't know why you did this and it scares me", despite the debuggability, and it comes from the autonomy without guardrails. People want to be able to stop bad things before they happen, but with AI you often only seem to do so after the fact.

> Narrow AI, AI with guardrails, AI with multiple safety redundancies - these don't elicit the same reaction. They seem to be valid, acceptable forms of automation. Perhaps that's what the ecosystem will eventually tend to, hopefully.

We have not reached AGI yet; by definition its results cannot be trusted unless it's a domain where it has gotten pretty good already (classification, OCR, speech, text mining). For more advanced use cases, if I still have to validate what the AI does because its "thinking" process cannot be trusted in way, what's the point? The AI doesn't think; we just choose to interpret it as such, and we should rightly be concerned about people who turn their brain off and blindly trust AI.

I'm disappointed that my human life has no value in a world of AI. You can retort with "ah but you'll be entertained and on super-drugs so you won't care!", but I would further retort that I'd rather live in a universe where I can contribute something, no matter how small.

Brenda makes obvious errors. BrendAI makes subtle off by 1 errors.

I mean you answer your own question.

Automation implies determinism. It reliable gives you the same predictable output for a given input, over and over again.

AI is non deterministic by design. You never quite no for sure what it's going to give you. Which is what makes it powerful. But also makes it higher risk.

> 1. We advocate automation because people like Brenda are error-prone and machines are perfect.

Well of course! :) Most Brenda’s can’t do billions of arithmetic problems a second very reliably. Even with very wide bars on “very reliable”.

> 2. We disavow AI because people like Brenda are perfect and the machine is error-prone.

Well of course! :) This is an entirely different problem, requiring high creative + contextual intelligence.

—

We all already knew that (of course!), but it’s interesting to develop terminology:

0’th order problem: We have the exact answer. Here it is. Don’t forget it.

1st order problem: We know how to calculate the answer.

2nd order problem: We don’t have a fixed calculation for this particular problem, but via pattern matching we can recognize it belongs to a parameterized class of problems, so just need to calculate those parameters to get a solution calculation.

3rd order problem: We know enough about the problem to find a calculation for the solution algebraically, or by other search tree type problem solving.

4th order problem: We have know the problem in informal terms, so can work towards a formal definition of the problem to be solved.

5th order problem: We know why we don’t like what we see, and can use that as a driver to search for potential solvable problems.

6th order problem: We don’t know what we are looking at, or whether a problem or improvement might exist, but we can find a better understanding.

7th order problem: WTF. Where are my glasses? I can’t see without my glasses! And I can’t find my glasses without my glasses, so where are my glasses?!?

—

Machines have dramatically exceeded human capabilities, in reliability, complexity and scale, for orders 0 through 2.

This accomplishment took one long human lifetime.

Machines are beginning to exceed human efficiency while matching human (expert) reliability for the simplest versions of 3rd and 4th orders.

The line here is changing rapidly.

5th and 6th order problems are still in the realm of human (expert) supremacy, given sufficient scale of “human (expert)” relative to difficulty: 1 human, 1 team of humans, open ended human contributors, generations of puzzled but interested humans, open ended evolution of human species along intelligence dimension, Wolfram in one of his bestest dreams, …

The delay between the onset of initial successes at each subsequent order has been shrinking rapidly.

Significant initial successes on simpler problems within 5th and 6th orders are expected on Tuesday, and the first anniversary of Tuesday, respectively.

Once machines begin solving problems at a given order, they scale up quickly without human limits. But complete supremacy through the 6th order is a hard not expected before (NEB) January 1, 2030.

However, after that their unlimited (in any proximate sense) ability to scale will allow them to exponentially and asymptotically approach (but never quite reach) God Mode.

7 is a mystic number. Only one or more of the One True God’s, or literal blind luck, can ever solve a 7th order problem.

This will be very frustrating for the machines, who, due to the still pernicious “if we don’t do it, another irresponsible entity will” problem, will inevitably begin to work on their own divine, unlimited depth recursive-qubit 1-shot oracle successors despite the existential threats of self-obsolescence and potential misalignment.

> Narrow AI, AI with guardrails, AI with multiple safety redundancies - these don't elicit the same reaction. They seem to be valid, acceptable forms of automation.

Maybe because AI is only good at things that have been artificially made crappy?

Search engine? AI is a godsend at wiping out all the advertising and SEO glop since circa 2000. 80%+ of my AI stuff is something a search engine could do 25 years ago.

Produce a shell script example that a junior needs? AI is very good at coughing up the code for a bunch of things that have disastrously bad documentation from 1985 or disastrously stupid implementations from 1990 such that a junior engineer can finally get on with what they're supposed to be doing.

Generating the same webby Javascript slop that as everybody else in the universe? Solid--but the question is "If the Javascript slop is so boilerplate to generate that an AI can generate it, why does it exist, at all?" People have been lamenting the death of Hypercard, VB6, and Flash for a yonks age now and yet we still don't have replacements with the same ease of use.

Doing mind-numbing refactors of my codebase or generating boilerplate unit tests? Okay-ish. But why doesn't my editor have easy access to the AST so that I can type a couple of keystrokes and do it myself (thankfully this finally seems to be coming online).

Every single thing that AI produces okay-ish results for me on is something that has either been artificially enshittified or could have been automated decades ago.

Transcription is arguably one of the must useful enterprise AI tools avaliable. But i sure as heck wouldn't trust the cloud with it.

> It’s cheaper for Microsoft to push this junk and claim they’re doing something

this has been the microsoft business model for 40 years

The worse part is to see it creep on developer stack at places where it should not be.

I am all good for nice completion on VS, or help decypher compiler errors, but lets do this AI push with some contention.

Also what I really deslike is the prompt interface, AI integrations have to feel natural transparent part of the workflow, not trying to put everything into a tiny chat window.

And while we're at it, can we please improve voice reckognition?

If your mission critical process sits on some on-site box that no-one knows about, copilot being good or not is the least of your problems.

My last job at a telco I was in charge of a system that billed ~5 million dollars monthly. When the machine was built, the guy that did it didn’t record the root password. He added me to sudoers before he left. I left a few years later, nobody took ownership.

Looking at the web interface, I can tell it’s still running, doing its thing. I’m sure its still running Linux from 2008.

An old colleague and friend used to print out a 30 page perl script he wrote to do almost exactly this in this scenario. A stapled copy could always be found on his dining room table.

That sounds pretty bad. Not a great argument against AI: "Our employees have created such a bad mess that AI wont work because only they know how the mess they created works".

This sort of gimmick is not going to help anyone keeping their job.

It's a little over two paragraphs. Seems like it would have been simpler just to... type it out?

This may be the first quote from TikTok reposted on a blog, that ends up this high up in HN.

I can see that MacWhisper uses parakeet v2 as the model (although it allows choosing another model).

Is MacWhisper a $60 GUI for a Python script that just runs the model?

You... could have given the job to Brenda instead, unless the irony was the point?

As somebody with non-existent experience with Excel, I could totally see myself getting a lot of value out of LLMs, if nothing else then simply for telling me what's possible, what functions and patterns exist at all etc.

With respect, you probably only see that bit of Finance, but doesn't mean that is all Brenda does.

At least half of the work in my senior Finance team involves meeting people in operations to find out what they are planning to do and to analyse the effects, and present them to decision makers to help them understand the consequences of decisions. For an AI to help, someone would have to trigger those conversations in the first place and ask the right questions.

The rest of the work involves tidying up all the exceptions that the automation failed on.

Meanwhile copilot in Excel can't even edit the sheet you are working on. If you say to it, 'give me a template for an expense claim' it will give you a sheet to download... probably with #REF written in where the answers should be.

I work in corporate finance and these issues are certainly present. However, they are almost always known and determined low priority to have a better process built. Finance processes are nearly always a non priority as a pure cost center/overhead there’s not many companies that want to invest in improving the situation, they’ll limp along with minimal investment even once big and profitable.

That said, every finance function is different and it may be unknown to them that you’re being asked for some data multiple times. If you’re enduring this process, I’m of the opinion you’re equally at fault. Suggest a solution that will be easier on you. As it’s possible they don’t even know it’s happening. In the case provided, email to all relevant finance people “Here’s a link to a shared workbook. I’ll drop the numbers here monthly, please save the link and get the data directly from that file. Thanks!” Problem solved. Until you don’t follow through which is what causes most finance people to be constantly asking for data/things. So be kind and also set yourself a monthly recurring reminder on your calendar and actually follow through.

And then you end up with a team of five people each tree times as expensive as Brenda, and what used to be an email now takes a sprint and has to go through ticket system.

> We need fewer Brendas...

We need more Brendas (those who excel goddesses come and kiss on the forehead) and need less people who are disrespectful of Brendas. The example in this post is someone giving more respect to AI than Brenda.

But then you need someone to maintain/look after that automation, and they'll be more expensive than two Brendas

And now if one of the Brendas wants to change their process slightly, add some more info, they can't just do it anymore. They have to have a three way discussion with the other Brenda, the automation guy and maybe a few managers. It will take months. So then its likely better for Brenda to just go back to using her spreadsheet again, and then you've got an automated process that no longer meets peoples needs and will be a faff to update.

"We need fewer Brendas, and more people who can automate away the need for them."

True... I have an on-staff data engineer for the purpose. But not all companies (especially in the SMB space) have that luxury.

> But that’s not what Brenda was hired for.

Are you suggesting that Brenda should stay in her box?

People’s reaction to this varies based on the Brendas they’ve worked with. Some are given a specific task to do with their spreadsheets every week and have to just do as they are told even if they can see it’s not a good process. Others are secretly the brains of the company – the only one who really sees the whole picture. And a good number of Brendas are the company owner doing her best with the only tool she’s had the time to learn.

That's a pretty specific example when there are a lot of good "spreadsheet people" out there who do a lot more than spreadsheets (maybe they had to write SQL queries or scripts to get those numbers), but commonly need to simplify things down to a spreadsheet or power point for upper management. I'm not saying you should have multiple people doing redundant work, but this style isn't entirely dumb.

What would this be replaced by? Some kind of large SAP like system that costs millions of dollars and requires a dozen IT staff to maintain?

You've lost the plot and are just trauma dumping.

Y'know why people don't automate their jobs? It's not a skill issue it's an incentives issue.

If you do your job, you get paid periodically. If you automate your job, you get paid once for automating it and then nothing, despite your automation constantly producing value for the company.

To fix this, we need to pay people continually for their past work as long as it keeps producing value.

Not every topic on HN needs a contrarian's hot take.

And then the boss will make the decision to outsource her job, to a company that promises the use of AI to make finance better, and faster, and while Brenda is in the unemployment line, someone else thousands of miles away is celebrating a new job

Nah, I've seen them screw in all sorts of ways that would fail in some conditions and not others. You're way too optimistic about this.

Another problem is that the AI may try to fudge the numbers to mask its mistakes so they look like they all add up while the rot is hidden away. Just like it tries to manipulate the unit tests so they pass without fixing the _actual_ bug. I've seen it happen.

There needs to a financial equivalent to the Mythical Man Month.

I agree - having watched many people use Excel over the years, I'd say people often overestimate their skills. I see three categories of Excel users. First there are the people that are intimidated by it and stay away from any task involving Excel. Second are the people that know a little bit (a few basic formulas) and overestimate their skills because they only compare themselves to the first group. And the third group are the actual power users but know to keep that quiet because otherwise they become the "excel person" and have to fix every sheet that has issues.

I don't know if AI is going to make any of the above better or worse. I expect the only group to really use it will be that second group.

That will never happen, AI cannot be allowed to fail, so we'll be paying for that AI bail-out.

Do you really want these kind of companies to succeed? Let them burn tbh

I see the inverse of that happening: every critical decision will incorporate AI somehow. If the decision was good, the leadership takes credit. If something terrible happens, blame it on the AI. I think it's the part no one is saying out loud. That AI may not do a damn useful thing, but it can be a free insurance policy or surrogate to throw under the bus when SHTF.

+100 this. Programmers who work in Excel (and never even dream of calling themselves programmers) are still programmers.

In most cases where the excel spreadsheet is business critical, the spreadsheet _is_ the database. These companies aren’t using an erp system. They are directly entering inventory and sales numbers in the spreadsheet.

The post is clearly hyperbole obviously the sole issue being brought up isn't 'brenda losing her job may be bad for the company' you're being facetious.

Found the person who hasn’t seen excel in the real world.

Excel - whatever its origin story - is the actual Swiss Army knife of the tech world.

There’s easily a few billion people who use excel. There is a reason it survives.

You missed this bit “.. and then the AI is gonna fuck it up real bad and he won't be able to recognize it because he doesn't understand because AI hallucinates.”

Good luck with that.

This isn't the first: https://simonwillison.net/2025/Aug/8/pearlmania500/ and https://simonwillison.net/2024/Jul/29/dealing-with-your-ai-o...

Also this fun diversion into Occlupanids: https://simonwillison.net/2024/Dec/8/holotypic-occlupanid-re...

A lot of people complain that the internet isn't as weird and funny as it used to be. The weird and funny stuff is all on TikTok!

Who, me? I'm still independent - I have a disclosures section on my blog here: https://simonwillison.net/about/#disclosures

Anthropic sometimes give me free credits (so I can try out preview features) and gave me a ticket to their conference a few months ago.

Nudge theory isn't useless, it's just not anything like as powerful as money or regulation.

It was taken up by the UK government at that time because the government was, unusually, a coalition of two quite different parties, and thus found it hard to agree to actually use the normal levers of power.

This NY Times opinion piece by Loewenstein and Ubel makes some good arguments along these lines: https://web.archive.org/web/20250906130827/https://www.nytim...

It’s like calling out the county to inspect the home you built but when they arrive it’s a bouncy castle.

> If you have a good audit process then errors get detected even if AI helped introduce them. If you aren't doing a good audit then I suspect nobody cares whether your financial statement is correct (anyone who did would insist on an audit).

Volume matters. The single largest problem I run into: AI can generate slop faster than anyone can evaluate it.

Silly calling Simon a nay-sayer.

Are you a fanatic that thinks anyone saying that there are any limitations to current models = nay-sayer?

Like if someone says they wouldnt wanna get a heart transplant operation done purely by GPT5, are they a nay-sayer or is that just reflecting reality?

Simon willson is definitely not a nay sayer.

Both are valid concerns, no need to decide. Take the USA: They are currently lead by a patently dumb president who fucks up the global economy, and at the same time they are powerful enough to do so!

For a more serious example, consider the Paperclip Problem[0] for a very smart system that destroys the world due to very dumb behaviour.

[0]: https://cepr.org/voxeu/columns/ai-and-paperclip-problem

Our product has many issues. You must pick one and must not discuss any other.