I'm Jeff, one of the founders of this+that. I've worked on named entity detection, fact extraction, knowledge management and NLP throughout my career, starting in the NLP stone age of the 1990s. I wanted to see whether a model could now write a useful wiki straight from a company's email.

Enron's mail is an obvious test because it's public, so anyone can check the pages against the messages and the history. We used 45,047 emails from eight mailboxes. Every fact quotes the sentence it came from, and a fact expires on the date the mail shows it stopped being true.

The hardest part was that the model already knows Enron. It kept adding details from the public record that the email didn't contain. The methodology page shows how we caught those, what it still gets wrong and what it cost: about $11,000 at list prices (though we used a subscription), with no attempt to bring that down.

Happy to answer questions.

▲

radlad 2 hours ago | parent | next [-]

Your comment was dead, and many of your other comments are dead (some flagged.) I vouched for this one, but you may be shadow banned, or users may be picking up on LLM tendencies in your text and flagging them.

▲

jreynar an hour ago | parent | next [-]

Thanks, really appreciate you bringing it back from the dead and hope it provides some helpful context that didn't fit naturally into the web site.

	▲	BalinKing an hour ago \| parent [-]
		AI-generated or -edited comments are disallowed on HN, see https://news.ycombinator.com/newsguidelines.html.

▲

Vaushite 2 hours ago | parent | prev [-]

Hacker News automatically flags comments with LLM tendencies.

▲

grey-area an hour ago | parent | prev | next [-]

This is interesting and applicable to so many domains where people want to query a model about a dataset, and have it summarise or otherwise analyse that dataset.

One of the biggest problems about LLMs as machines for analysis is that they don't respect the boundaries of a problem and instead generate base on their corpus and training as well as the actual dataset in question in the context.

It would be wonderful if we could easily strip out facts about Enron somehow from a model or just instruct it to ignore anything outside the dataset (and have the instruction respected), it is quite frustrating that these machines are so fuzzy.

I had a look at the methodology page, well done for doing the hard work of going over the results a few times to find errors/inconsistencies. There are not too many inconsistencies according to that.

Of 62,205 candidate facts, 37 quoted a sentence we couldn't find in the message

https://www.thisandthat.chat/enron/methodology.html

The difficulty for a business considering using this (in law for example or analysing internal business documents), is that one incorrect fact could be very damaging and it's very hard to check them all.

	▲	jreynar 27 minutes ago \| parent [-]
		I completely agree. It's hard to imagine this ever being perfect both because of AI and because people make mistakes in communications all the time. In our own usage of it we've found it essential to check things that seem suspicious once they're in the company brain. But, the more important gate, and one we expect may remain longer than people would like, is that we ask people to approve changes to a company brain that are proposed. Even this isn't perfect but it is a key defense from the brain turning into slop. And it's different than this demo, which was much closer to fully automated.

▲

number6 2 hours ago | parent | prev | next [-]

Maybe someone in Germany needs your help with the AFD Email leak; right now journalist are going through the data; your solution sounds like something that need

▲

fwip an hour ago | parent | prev [-]

No, "you" are an LLM pretending to be Jeff, possibly at Jeff's behest.