Remix.run Logo
conradfr 6 hours ago

Call me crazy but sometimes I still find a better solution on StackOverflow than what Claude Code insists to do.

I'm not sure we're better off without SO in the long run.

gpugreg 6 hours ago | parent | next [-]

Same here. LLMs are great at spitting out well-known solutions to problems instead of the best one. The "long tail" of solutions is usually lost due to how tokens are sampled from the LLM's probability distribution.

What I found to help a lot is to ask for e.g. 10 different solutions to a problem and then choosing one of them. Sometimes, this even leads to borderline creative solutions if there aren't 10 different ones.

btown 6 hours ago | parent | next [-]

In theory reasoning tokens should do the equivalent of this - explicitly create options outside of the quick-response probability space, so those can guide future generation.

In practice, models that do this won't be prioritized as much, because the economics of thinking tokens that stop by default at, say, one option plus a bit more planning (short of full alternatives) would be superior as long as billing is per-user instead of per-token. So we'll still need to play games with prompting!

tliltocatl 6 hours ago | parent [-]

Without continuous feedback from real world, lower-probability token (and soon high-probability ones as well) will be complete garbage.

exe34 6 hours ago | parent | prev [-]

> LLMs are great at spitting out well-known solutions to problems instead of the best one.

I remember how Stack Overflow would close questions as duplicates just because somebody suggested the wrong answer that is also the right answer to the existing question. The best way to get a correct answer on Stack Overflow (and forums before that) was to post the wrong answer as part of your question.

Morromist 6 hours ago | parent | prev | next [-]

One thing that SO had was you could see multiple solutions and implementations for something. Sometimes the "best" solution isn't very readable code, sometimes you are able to understand the problem better when you see a bunch of people solving it in different ways and arguing about it like angry monkeys.

It really could be bad though.

ssl-3 6 hours ago | parent | next [-]

The bot can do that kind of thing, too.

"Show me 6 very different solutions, and present arguments for/against each one as if a bunch of angry monkeys."

allknowingfrog 6 hours ago | parent | prev [-]

SO has always had a pretty strong stance against opinion-based questions, but this is maybe the niche they should be exploring now. Humans still have a lot to say about the "best" solution to a given problem. The whole idea of an "accepted" answer could be removed, for example, since that's what AI will already generate.

ceejayoz 6 hours ago | parent | prev | next [-]

Much of what Claude insists you do probably came from SO.

akkad33 6 hours ago | parent | next [-]

Or Reddit. I don't know about Claude but Gemini has given me answers that are verbatim comments from Reddit.

arcanemachiner 6 hours ago | parent | next [-]

I've gotten my own answers given back to me for problems I forgot I already had.

mcswell an hour ago | parent | next [-]

Somebody posted a similar comment above yours (somewhere...). I don't think your experience is unique!

irishcoffee 6 hours ago | parent | prev [-]

I had email correspondence once with a vendor about how to talk to their i2c bus. The documentation was all asm, and I wanted to at least “uplift” to C. They didn’t have any answers, so I sent them my solution which was was the asm calls that the c stdlib decompiled into.

4 years later my company had bought a different company, who happened to be using a newer model of the same board. They asked me how we could use the 12c bus. “Well before you bought us, we emailed the vendor and sent back this C snippet”

It was my code, verbatim. I’ve always wondered how many times they passed that bit of code around.

dd8601fn 6 hours ago | parent | prev | next [-]

Claude does it quite a bit when you’re triggering the search tool functions.

It’s fine, and what you would expect for certain prompts, except that the synthesized results often come back communicating more authority than they deserve.

lukan 5 hours ago | parent [-]

It was funny for me, when I asked it about something specific exotic - and it gave me a confident answer. But checking the sources I discovered it was from my own inquiries on a forum thread about it from the last time I unsuccesfully tried this (before the agents came) And so I knew, that any authorative tone was undeserved.

On the other hand, Claude later nailed this project, where I as a human said before, no, too much extra work.

Morromist 6 hours ago | parent | prev | next [-]

I've gotten this too a lot. If you ask AI to cite where it got info you can lose a lot of confidence in it pretty quickly.

worthless-trash 6 hours ago | parent | prev [-]

I have seen it quote my own code back at me, including comments word for word.

20k 6 hours ago | parent | prev [-]

I've seen chatgpt word for word plagiarise stack overflow answers

andrekandre 5 hours ago | parent [-]

i've seen it plagiarize personal blog posts too, almost verbatim code line by line.... kind of shocking...

tartoran 5 hours ago | parent [-]

Well, it's a plagiarizing machine after all and most of the time it remixes it well enough so most people can't tell.

Bender 2 hours ago | parent | prev | next [-]

I'm not sure we're better off without SO in the long run.

You're right but that site has been sputtering culturally for some time. I put a lot of effort into editing questions and answers on ServerFault (part of SO) but I feel that time was wasted. I think they knew for a while they just wanted to sell it and just stopped caring. A number of editors were allowed to be jerks for too long and it went to their heads. I wish I could take back all that effort.

raffael_de 6 hours ago | parent | prev | next [-]

well, SO is probably the highest quality data source for a language model and the rest of the internet is just diluting the final latent space limited by Jon Skeet.

whateverboat 6 hours ago | parent | prev | next [-]

What you are noticing in a long term is the "community" knowledge and communication which the chatGPT is now kind of destroying. In some sense, it is no different from the difference between studying along and studying with your peers at a university.

You can definitely study alone and achieve perfect grades, but studying with your peers is how you build relationships for future life and take your community forward as a whole.

andrekandre 5 hours ago | parent | prev | next [-]

was just gonna post the same thing

needed to implement a language feature that was a bit complicated and im not familiar with it so just planned with claude to do it, and after each write/fix cycle it just wouldn't work right.... gave up, went back to SO copy pasted the (not perfect but enough to start from) answer and worked up from there...

at the same time my knowledge grew and im more confident to do this same capability myself whereas reiterating with claude it was just a slog and i didn't learn much...

i think i may be starting to sour on these "do it all for me" usage scenarios for ai... especially for unfamiliar areas...

ImageXav 5 hours ago | parent | prev | next [-]

Agreed. Which is also odd, if you think about it. Surely with the amount of compute Anthropic and others have available, they could test each of the solutions in the SO data they surely have and rank them based on efficiency/elegance/other criteria and remove poor solutions from their training data.

sixtyj 6 hours ago | parent | prev | next [-]

I am thinking to make canned encyclopaedia of stackoverflow answers.

Claude/Grok/Gemini/Chatgpt answers are often so… how to say it… misleading? I have to stop the conversation as it leads nowhere (and it is not a skill issue :)

jshen 6 hours ago | parent | prev | next [-]

We may need to create a community driven version of SO. Hard for it to be a successful business these days.

asqueella 6 hours ago | parent [-]

https://software.codidact.com/ was created after one of the many SO dramas. It doesn't come up in searches though and I didn't have reason to use it...

jshen 5 hours ago | parent [-]

thanks, that's exactly what I was imagining.

tayo42 4 hours ago | parent | prev | next [-]

What are you looking for and finding on stack overflow that isn't begginer to intermediate level?

6 hours ago | parent | prev | next [-]
[deleted]
FrustratedMonky 6 hours ago | parent | prev | next [-]

And, the AI trained on Stack Overflow. So if no one is posting new questions, and new answers. What will AI train on next, for the next thing.

ishurand4 6 hours ago | parent | next [-]

Stack Overflow and Reddit are still getting threads. And as AI gets smarter, the questions will also expand.

bigfishrunning 6 hours ago | parent | prev [-]

your prompts, and the code you have it review.

FrustratedMonky 5 hours ago | parent [-]

Maybe.

I thought point was on Stack Overflow, there were community voting on 'best' answer.

If it is just me and the AI. Then the AI training data, is just whatever I approved the AI to do. Just my opinion.

deaton 6 hours ago | parent | prev [-]

Definitely not better off. SO was fairly mean spirited, but nowhere else has such a vast trove of high quality answers to common software problems been collected. SO likely trained many of these models with its answers, and I don't know what software development will look like when it dies.

NoMoreNicksLeft 6 hours ago | parent [-]

No where else has such a vast trove of high quality answers been hidden because the question was closed as duplicate when someone else asked that question later.