Nice work. Subscribed.

I had a very similar idea a while back. I wanted to rank news by "impact" which might be more concrete than "significance."

For an LLM prompt, it would be something like:

"estimate the number of people who's lives that will be materially changed by this news." and "estimate the average degree of change for those impacted."

Then impact is roughly the product of those two.

Additionally, I want a version that is tailored to me specifically "estimate the degree of change this will have on my life." + context of my life.

Tangentially, I've found that getting ratings out LLMs works better when I can give all options and request relative ratings. If I ask for rankings individually I get different and less good results. Not enough context length to rate all news from all time in one go though. Any thoughts on that? Maybe providing some benchmark ratings with each request could help? Something I'm exploring.

What you're describing is super close to the first version I had!

In the beginning I had 3 parameters: scale (number of people), magnitude (degree of change for those impacted) and additionally potential (how likely is this event to trigger downstream significant events).

The point behind including potential was to separate these two events:

1) A 80 year old dies from cancer 2) An 80 year old dies from a new virus called COVID

This worked roughly well but I kept adding parameters to improve the system: novelty, credibility, etc... The current system works on 7 parameters.

---

I never attempted to give LLM all options and rank them against each other.

1) as you said, for me 20k articles is just too much to fit into context window. Maybe some modern LLMs can handle it, but it wasn't the case for a long time, and I settled on current approach.

2) I don't want the "neighbors" to affect individual article ratings. With the current system I am able to compare news spread over months, because they were all rated using the same prompt.

3) I intentionally avoided giving AI examples, like "evaluate event X given that event Y is 7/10". I want it to give scores with a "clear mind" and not be "primed" to my arbitrary examples.