Now I'm very interested in seeing the transcript/session from what the agent actually did.

We already know since before that agents takes all sorts of possible shortcuts to find the answer they think the user was looking for. It doesn't seem impossible to me that the user asked something that made the agent feel OK with highlighting something that wasn't logged in the Mexican database as a novel discovery, even if it wasn't.

But again, transcripts/session logs would make it extremely clear exactly what has happened, and I'd take the author not publishing those as trying to hide something about how the agent actually approached this.

▲

letmevoteplease a day ago | parent [-]

"I'd take the author not publishing those as trying to hide something about how the agent actually approached this."

That is an irrationally cynical take. Everything described in the post is clearly within the abilities of a frontier LLM. The previous discovery is obscure enough that the Bluesky author is not citing any actual publication but instead encouraging people to reach out to a scientist and receiving images through private communication. The reasonable interpretation is that the model stumbled upon a previous discovery in the manner described, and the author/LLM failed to find the previous discovery because it is not any published database. The author is not running a scam for some unclear benefit that would be immediately revealed when he published it.

▲

embedding-shape a day ago | parent [-]

> That is an irrationally cynical take. Everything described in the post is clearly within the abilities of a frontier LLM.

No, the burden of proof that it's actually novel sits on the author of this post, and we assume everything is cheated/shortcutted by default when it comes to agents and LLMs, unless we can prove it actually "found" this, not just looked up stuff that wasn't properly in the DB yet but already written about.

> The reasonable interpretation is that the model stumbled upon a previous discovery in the manner described

That'd be reasonable indeed. Instead the author chose the headline "My AI Agents Found an Unmapped Maya Site" and the first sentence is:

> By analyzing NASA lidar data, my AI agents discovered a Maya site in Mexico that, as far as anyone can tell, no one had reported before

Which without any evidence of what the agent actually did, I don't think we should just assume what the author says is true because again, we know that agents takes as many shortcuts as they can.

Plenty of times the people pushing LLMs are over-exaggerating more than in other ecosystems, and the community should rightly push back on these things, when they're not presented properly with proof and evidence. Otherwise we'd be doing previous actual archeologists a huge disservice.

Imagine being the archeologist who actually discovered this site, and waking up today reading some random person taking credit for your discovery! Kind of feels like people here often forget there are other people involved on the other side of things.

▲

fancyfredbot a day ago | parent [-]

He checked with the Mexican government before claiming it was new, which is reasonable due diligence surely?!

▲

embedding-shape a day ago | parent [-]

No, you don't just do the simplest possible thing and see if it's in some DB because it happens to be more accessible to LLMs, then call it a day and a call it a "discovery". More from the Bluesky thread:

> any archaeologist knows to triple check before saying something is "new"

There is lots of due diligence needed before proclaiming stuff like that. Not just in archeology, but in sciences in general. There is an expectation that you'll review what you put out carefully, not rush to publish as soon as possible, and review prior art. Not least because it's embarrassing as hell to accidentally claim to have invented/discovered something, but in reality someone else was clearly way earlier than you, you just didn't look closely/broadly enough before.

As a second thing, once you find out you're wrong, you generally want to retract something to save face, if you're a good enough human to be able to admit when you're wrong. The author's "What I've learned" remains "You can just do things" even though the top of the article now has a correction that the agents maybe "missed it while going through the literature".

▲

chasd00 a day ago | parent [-]

>> any archaeologist knows to triple check before saying something is "new"

difficulty: not an archaeologist but a SWE

cut the guy/gal some slack

	▲	embedding-shape a day ago \| parent [-]
		> difficulty: not an archaeologist but a SWE cut the guy/gal some slack Maybe don't publish archaeology "findings" and "discoveries" if you're merely a software developer. I don't go around publishing things about nuclear reactors, because it's not my area, I couldn't even review myself if it's right or not.