This is cool - just wanted to note https://marginlab.ai is one that has been around for a while.

are there any tools anyone knows to collect this kind of telemetry while using the tools instead of offline evals.

running evals seems like it may be a bit too expensive as a solo dev.