The hard part is that it locks you in to whatever people believe at the moment, which might be proven wrong in the future

How would you design the pipeline and user interface to reflect the temporal nature and potential theta decay of evaluations? Reprocess comments on a cadence?

For example, however you represent a comment, it could then support expansion of a tree to each evaluation snapshot point in time, like expanding a directory listing its files. My thought on this is somewhat similar to Wayback Machine’s interface per URL; the URL remains static over time, with a calendar and indicators for date of snapshot, and drill down into each snapshot.

In this use case, the comment is the URL, and each LLM evaluation of the comment is a snapshot. Append only.