Very reminiscent of my experience pointing persistent agents at computational physics problems. It can find amazing results and really surprise me, make a write up, and convince me it's a real discovery... until i tell it to do a literature search, and be meticulous. Then it comes back saying "prior art has reduced the novelty of the discovery", and finds papers explaining the discoveries very easily and intelligently.

Perhaps google SWE should have requested a literature search before publishing his results as novel

▲

m_kos 20 hours ago | parent | next [-]

A year or two ago, I wasted over a month on side projects that I thought were novel. Now, another LLM scrutinizes any idea in an area I don't know well ("I have just received this from a junior researcher on my team. Scrutinize it very critically. Don't hold back. This early stage is the best time for honest, harsh feedback. Don't forget to include the analysis of X, Y, Z in your report.").

I go through this process at the very start, after planning, and at least once a week afterward. With AI, the landscape can change dramatically during one week. I am not being hyperbolic.

	▲	hirako2000 18 hours ago \| parent [-]
		Asking an LLM for criticism is like asking a yes-man to punch you in the face. he'll always swing, but hitting a straw man he built himself.

▲

jgilias a day ago | parent | prev | next [-]

A fancy autocomplete still. Said with no intent to disparage. If the prior art hasn’t been pulled into the context window, it’s not surprising for the completion to not take it into consideration.

▲

FartyMcFarter a day ago | parent [-]

Sometimes I feel like a fancy autocomplete myself.

Most LLMs these days can use tools (such as web searching), so depending on how strictly you define "autocomplete" this alone could disqualify it from being an autocomplete, fancy or otherwise.

▲

jgilias a day ago | parent [-]

For sure yeah! What I’m pointing out is that I see people mistaking LLMs for “truth engines”, when they’re not. I find the “autocomplete” mental model really useful for when I work with LLMs myself. E.g., always keep in mind what’s in context, what’s not, where to get context.

Goes without saying - humans are no truth engines either lol.

	▲	voidhorse 12 hours ago \| parent [-]
		Yes, "predictors" or "completers" are a far more accurate metal model for what they actually do and it pays to understand how they "use tools" and what RL, SFT, etc. is and how it makes LLMs seem like much more than what they still, ultimately, are. People latched onto the bs marketing terms like "thinking" way too easily. I don't know where all the classic hacker skepticism and desire to know how things actually work went the past four years.

▲

ebertucc a day ago | parent | prev | next [-]

I enjoy the contrite rationalizations and turns of phrase LLMs use when encountering a discrepancy.

"Prior art has reduced the novelty of the discovery" is a great one

	▲	idiotsecant a day ago \| parent [-]
		Maybe LLMs are getting a little too good at emulating humans.

▲

samudrijan a day ago | parent | prev [-]

It’s like a shell game. It’s likely overfitting on its training data.