Remix.run Logo
simianwords 10 hours ago

i don't get what the point of what you are saying is? i can ask it to explain how to solve an integral right now with steps.

i can ask it to tell me how to write like a person X right now.

RobRivera 3 hours ago | parent | next [-]

Actually this is the crux and the nuance which makes discussing LLM specifics a pain in the general space.

If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.

Instead what you will receive is a text that follows a statistically derived most likely (in accordance to the perplexity tuning) response to such a question.

simianwords 22 minutes ago | parent | next [-]

>If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.

this shows that you have very less idea on how llm's work.

LLM that is trained only on john steinbeck will not work at all. it simply does not have the generalised reasoning ability. it necessarily needs inputs from every source possible including programming and maths.

You have completely ignored that LLMs have _generalised_ reasoning ability that it derives from disparate sources.

bigfishrunning 9 minutes ago | parent [-]

LLMs have the ability to convince you that they've "reasoned". sometimes, an application will loop the output of an LLM to its input to provide a "chain of reasoning"

This is not the same thing as reasoning.

LLMs are pattern matchers. If you trained an llm only to map some input to the output of John Steinbeck, then by golly that's what it'll be able to do. If you give it some input that isn't suitably like any of the input you gave it during training, then you'll get some unpredictable nonsense as output.

simianwords a minute ago | parent [-]

this is outdated stuff from 3 years ago.

> If you trained an llm only to map some input to the output of John Steinbeck

this is literally not possible because the llm does not get generalised reasoning ability. this is not a useful hypothetical because such an llm will simply not work. why do you think you have never seen a domain specific model ever?

if you wanted to falsify this claim: "llm's cant reason" how would one do that?

here is gpt-5.4 solving never before seen mathematics problems: https://epochai.substack.com/p/gpt-54-set-a-new-record-on-fr...

you could again say its just pattern matching but then i would argue that its the same thing we are doing.

netdevphoenix 3 hours ago | parent | prev [-]

> If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.

Isn't this obvious? There is not enough latent knowledge of math there to enable current LLMs to approximate anything resembling an integral.

RobRivera 3 hours ago | parent | next [-]

Its obvious to me.

Its obvious to you.

It isnt obvious to the person I am responding to, and it isnt obvious to majority of individuals I speak with on the matter (which is why AI, personally, is in the bucket of religion amd politics for polite conversation to simply avoid)

simianwords 29 minutes ago | parent | next [-]

It’s obvious to me. What point are you trying to make? It’s not religion it’s falsifiable easily.

LLMs can reason about integrals as well as in a literature context. You suggested that if it’s not trained on literature then it can’t reason about it. But why does that matter?

kenjackson an hour ago | parent | prev [-]

Wait -- I'm fairly certain this is obvious to the person you were responding to. It may not be obvious to a lay person (who may not even know LLMs are trained at all). But I think this is obvious to almost all people with even a small understanding of LLMs.

Talanes 3 hours ago | parent | prev [-]

Now what if we ask the LLM to write about social media? Do you think the output would be similar to what you'd get if we had a time machine to bring the actual man back and have him form his own thoughts firsthand?

bigfishrunning 8 minutes ago | parent [-]

It may be stylistically similar, but it's impossible to predict what the content would be.

Peritract 10 hours ago | parent | prev | next [-]

"Explain how to solve" and "write like X" are crucially different tasks. One of them is about going through the steps of a process, and the other is about mimicking the result of a process.

z2 10 hours ago | parent | next [-]

Neural networks most certainly go through a process to transform input into output (even to mimic the results of another process) but it's a very different one from human neutral networks. But I think this is the crucial point of the debate, essentially unchanged from Searle's "Chinese Room" argument from decades ago.

The person in that room, looking up a dictionary with Chinese phrases and patterns, certainly follows a process, but it's easy to dismiss the notion that the person understands Chinese. But the question is if you zoom out, is the room itself intelligent because it is following a process, even if it's just a bunch of pattern recognition?

simianwords 10 hours ago | parent | prev [-]

but llm can do both. so what's the point?

can you give a specific example of what an llm can't do? be specific so we can test it.

plewd 10 hours ago | parent [-]

like OP originally said, the LLM doesn't have access to the actual process of the author, only the completed/refined output.

Not sure why you need a concrete example to "test", but just think about the fact that the LLM has no idea how a writer brainstorms, re-iterates on their work, or even comes up with the ideas in the first place.

JCharante 2 hours ago | parent | next [-]

why not? datasets are not only finished works, there's datasets that go into the process they're just available in smaller quantities

elmomle an hour ago | parent [-]

Let's take the work of Raymond Carver as just one example. He would type drafts which would go through repeated iteration with a massive amount of hand-written markup, revision and excision by his editor.

To really recreate his writing style, you would need the notes he started with for himself, the drafts that never even made it to his editor, the drafts that did make to the editor, all the edits made, and the final product, all properly sequenced and encoded as data.

In theory, one could munge this data and train an LLM and it would probably get significantly better at writing terse prose where there are actually coherent, deep things going on in the underlying story (more generally, this is complicated by the fact that many authors intentionally destroy notes so their work can stand on its own--and this gives them another reason to do so). But until that's done, you're going to get LLMs replicating style without the deep cohesion that makes such writing rewarding to read.

mold_aid 19 minutes ago | parent | next [-]

A good point. "Famous author" is a marketing term for Grammarly here; it's easy to conceive of an "author" as being an individual that we associate with a finite set of published works, all of which contain data.

But authors have not done this work alone. Grammarly is not going to sell "get advice from the editorial team at Vintage" or "Grammarly requires your wife to type the thing out first, though"

I'll also note that no human would probably want advice from the living versions of the author themselves.

simianwords 27 minutes ago | parent | prev [-]

Can a human replicate style without understanding process? Yes we can. We do it all the time with Shakespeare. Why not LLMs?

I can do it at the moment with Shakespeare an LLMs.

empath75 3 hours ago | parent | prev | next [-]

> has no idea how a writer brainstorms

This isn't true in general, and not even true in many specific cases, because a great deal of writers have described the process of writing in detail and all of that is in their training data. Claude and chatgpt very much know how novels are written, and you can go into claude code and tell it you want to write a novel and it'll walk you through quite a lot of it -- worldbuilding, characters, plotting, timelines, etc.

It's very true that LLMs are not good at "ideas" to begin with, though.

GCA10 2 hours ago | parent | next [-]

Professional writer here. On our longer work, we go through multiple iterations, with lots of teardowns and recalibrations based on feedback from early, private readers, professional editors, pop culture -- and who knows. You won't find very clear explanations of how this happens, even in writers' attempts to explain their craft. We don't systematize it, and unless we keep detailed in-process logs (doubtful), we can't even reconstruct it.

It's certainly possible to mimic many aspects of a notable writer's published style. ("Bad Hemingway" contests have been a jokey delight for decades.) But on the sliding scale of ingenious-to-obnoxious uses for AI, this Grammarly/Superhuman idea feels uniquely misguided.

AlotOfReading 2 hours ago | parent | prev | next [-]

The distinction being made is the difference between intellectual knowledge and experience, not originality.

Imagine a interviewing a particularly diligent new grad. They've memorized every textbook and best practices book they can find. Will that alone make them a senior+ developer, or do they need a few years learning all the ways reality is more complicated than the curriculum?

LLMs aren't even to that level yet.

re-thc 3 hours ago | parent | prev [-]

> because a great deal of writers have described the process of writing in detail

And that's often inaccurate - just as much as asking startup founders how they came to be.

Part of it is forgot, part of it is don't know how to describe it and part of it is don't want to tell you so.

10 hours ago | parent | prev | next [-]
[deleted]
simianwords 10 hours ago | parent | prev [-]

i don't buy this logic. if i have studied an author greatly i will be able to recognise patterns and be able to write like them.

ex: i read a lot of shakespeare, understand patterns, understand where he came from, his biography and i will be able to write like him. why is it different for an LLM?

i again don't get what the point is?

wongarsu 10 hours ago | parent | next [-]

You will produce output that emulates the patters of Shakespeare's works, but you won't arrive at them by the same process Shakespeare did. You are subject to similar limitations as the llm in this case, just to a lesser degree (you share some 'human experience' with the author, and might be able to reason about their though process from biographies and such)

As another example, I can write a story about hobbits and elves in a LotR world with a style that approximates Tolkien. But it won't be colored by my first-hand WW1 experiences, and won't be written with the intention of creating a world that gives my conlangs cultural context, or the intention of making a bedtime story for my kids. I will never be able to write what Tolkien would have written because I'm not Tolkien, and do not see the world as Tolkien saw it. I don't even like designing languages

simianwords 9 hours ago | parent [-]

that's fair and you have highlighted a good limitation. but we do this all the time - we try to understand the author, learn from them and mimic them and we succeed to good extent.

that's why we have really good fake van gogh's for which a person can't tell the difference.

of course you can't do the same as the original person but you get close enough many times and as humans we do this frequently.

in the context of this post i think it is for sure possible to mimic a dead author and give steps to achieve writing that would sound like them using an LLM - just like a human.

Peritract 8 hours ago | parent [-]

You're still confusing "has a result that looks the same" and "uses the same process"; these are different things.

simianwords 6 hours ago | parent [-]

Why do you say it has a different process? When I ask it to do integrals it uses the same process as me

Peritract 5 hours ago | parent | next [-]

Not everything works like integrals. Some things don't have a standard process that everyone follows the same way.

Editing is one of these things. There can be lots of different processes, informed by lots of different things, and getting similar output is no guarantee of a similar process.

simianwords 3 hours ago | parent | next [-]

I don’t see why editing is any different. If a human can learn it why not an llm

esafak 3 hours ago | parent | prev [-]

The process is irrelevant if the output is the same, because we never observe the process. I assume you are arguing that the outputs are not guaranteed to be the same unless you reproduce the process.

If we are talking about human artifacts, you never have reproducibility. The same person will behave differently from one moment to the next, one environment to another. But I assume you will call that natural variation. Can you say that models can't approximate the artifacts within that natural variation?

Rohansi 2 hours ago | parent | next [-]

It's relevant for data it hasn't been trained on. LLMs are trained to be all-knowing which is great as a utility but that does not come close to capturing an individual.

If I trained (or, more likely, fine-tuned) an LLM to generate code like what's found in an individual's GitHub repositories, could you comfortably say it writes code the same way as that individual? Sure, it will capture style and conventions, but what about our limitations? What do you think happens if you fine-tune a model to write code like a frontend developer and ask it to write a simple operating system kernel? It's realistically not in their (individual) data but the response still depends on the individual's thought process.

simianwords 7 minutes ago | parent | next [-]

>If I trained (or, more likely, fine-tuned) an LLM to generate code like what's found in an individual's GitHub repositories, could you comfortably say it writes code the same way as that individual? Sure, it will capture style and conventions, but what about our limitations? What do you think happens if you fine-tune a model to write code like a frontend developer and ask it to write a simple operating system kernel? It's realistically not in their (individual) data but the response still depends on the individual's thought process.

Look, I don't think you understand how LLM's work. Its not about fine tuning. Its about generalised reasoning. The key word is "generalised" which can only happen if it has been trained on literally everything.

> It's relevant for data it hasn't been trained on

LLM's absolutely can reason on and conceptualise on things it has not been trained on, because of the generalised reasoning ability.

esafak an hour ago | parent | prev [-]

I don't know if LLMs are trained to imitate sources like that. I also don't know what would happen if you asked it to do something like someone who does not know how to do it. Would they refuse, make mistakes, or assume the person can learn? Humans can do all three, so barring more specific instructions any such response is reasonable.

volkk 44 minutes ago | parent | prev [-]

i think there's a lot to be said about the process as well, the motivations, the intuitions, life experiences, and seeing the world through a certain lens. this creates for more interesting writing even when you are inspired by a certain past author. if you simply want to be a stochastic parrot that replicates the style of hemingway, it's not that difficult, but you'll also _likely_ have an empty story and you can extend the same concept to music

arkadiytehgraet an hour ago | parent | prev [-]

Even if the visualization of the integration process via steps typed out in the chat interface is the same as what you would have done on paper, the way the steps were obtained is likely very different for you and LLM. You recognized the integral's type and applied corresponding technique to solve it. LLM found the most likely continuation of tokens after your input among all the data it has been fed, and those tokens happen to be the typography for the integral steps. It is very unlikely are you doing the same, i.e. calculating probabilities of all the words you know and then choosing the one with the highest probability of being correct.

simianwords 24 minutes ago | parent [-]

> the way the steps were obtained is likely very different for you and LLM

this is not true, any examples?

inaros 3 hours ago | parent | prev | next [-]

>> i again don't get what the point is?

The point is that you dont become Jimi Hendrix or Eric Clapton even if you spend 20 years playing on a cover band. You can play the style, sound like but you wont create their next album.

Not being Jimi Hendrix or Eric Clapton is the context you are missing. LLMs are Cover Bands...

tovej 10 hours ago | parent | prev | next [-]

You can understand his biography and analyses about how shakespeare might have written. You can apply this knowledge to modify your writing process.

The LLM does not model text at this meta-level. It can only use those texts as examples, it cannot apply what is written there to it's generation process.

simianwords 9 hours ago | parent [-]

no it does and what you said is easily falsifiable.

can you provide a _single_ example where LLM might fail? lets test this now.

tovej 4 hours ago | parent [-]

Yes, what I said should be falsifiable. The burden is on you to give me an example, but I can give you an idea.

You need to show me an LLM applying writing techniques do not have examples in its corpus.

You would have to use some relatively unknown author, I can suggest Iida Turpeinen. There will be interviews of her describing her writing technique, but no examples that aren't from Elolliset (Beasts of the sea).

Find an interview where Turpeinen describes her method for writing Beasts of the Sea, e.g.: https://suffolkcommunitylibraries.co.uk/meet-the-author-iida...

Now ask it to produce a short story about a topic unrelated to Beasts of the Sea, let's say a book about the moonlanding.

A human doing this exercise will produce a text with the same feel as Beasts of the Sea, but an LLM-produced text will have nothing in common with it.

simianwords 24 minutes ago | parent [-]

>You need to show me an LLM applying writing techniques do not have examples in its corpus.

why are you bringing this constraint?

TimorousBestie 10 hours ago | parent | prev [-]

This is the plot of a short story of Borges’ called “Pierre Menard, the Author of Don Quixote.”

mysterydip 9 hours ago | parent | prev [-]

Is the reason it can show steps for solving an integral because the training set contained webpages or books showing how to do it?

simianwords 9 hours ago | parent [-]

if we have steps for understanding any author's english and creative process (generally not specific to an author) would you agree then it is possible for an llm to do it?

Talanes 3 hours ago | parent | next [-]

The real sticking point for me is I don't even believe that authors themselves FULLY understand their process. The idea that anybody could achieve such full introspection as to understand and articulate every little thing that influences their output seems astoundingly improbable.

mysterydip 8 hours ago | parent | prev [-]

Repeating a process, yes for sure, even (pseudorandom?) variations on a process. Understanding a process is a different question, and I’m not sure how you would measure that.

In school we would have a test with various questions to show you understand the concept of addition, for example. But while my calculator can perfectly add any numbers up to its memory limit, it has no understanding of addition.

netdevphoenix 3 hours ago | parent | next [-]

> while my calculator can perfectly add any numbers up to its memory limit, it has no understanding of addition.

"my calculator can perfectly add any numbers up to its memory limit" This kind of anthropomorphic language is misleading in these conversations. Your calculator isn't an agent so it should not be expected to be capable of any cognition.

simianwords 6 hours ago | parent | prev [-]

It’s the degree of generalisability. And LLMs do have understanding. You can ask it how it came up with the process in natural language and it can help - something a calculator can’t do.

bigfishrunning 3 minutes ago | parent [-]

> And LLMs do have understanding.

They absolutely do not. If you "ask it how it came up with the process in natural language" with some input, it will produce an output that follows, because of the statistics encoded in the model. That output may or may not be helpful, but it is likely to be stylistically plausible. An LLM does not think or understand; it is merely a statistical model (that's what the M stands for!)