Remix.run Logo
lelanthran 3 hours ago

Iteration only matters when the feedback is used to improve.

Your model doesn't improve. It can't.

baq 3 hours ago | parent | next [-]

The magic of test time inference is the harness can improve even if the model is static. Every task outcome informs the harness.

thenaturalist 2 hours ago | parent [-]

> The magic

Hilarious that you start with that as TAO requires

- Continuous adaptation makes it challenging to track performance changes and troubleshoot issues effectively.

- Advanced monitoring tools and sophisticated logging systems become essential to identify and address issues promptly.

- Adaptive models could inadvertently reinforce biases present in their initial training data or in ongoing feedback.

- Ethical oversight and regular audits are crucial to ensure fairness, transparency, and accountability.

Not much magic in there if it requires good old human oversight every step of the way, is there?

mountainriver 2 hours ago | parent | prev [-]

Your model can absolutely improve

thenaturalist 2 hours ago | parent [-]

How would that work out barring a complete retraining or human in the loop evals?