Ok, so looking at the commit log[1], I was mostly interested in seeing what the "moonshot ideas" implementations looked like, but basically everything is just hyperparameter tuning. Which is nice, but likely not worth the $$$ spent on the tokens. Am I missing something here?

[1] https://github.com/ykumards/eCLIP/commits/main/autoresearch

▲

DoctorOetker 15 hours ago | parent | next [-]

It would seem wise to modify the autoresearch instructions to first estimate the computational costs rigorously and then sort and compare the proposals for human review, and for each actually executed attempt to feed back the computational costs with LoRa adapter?

i.e. perhaps minimal changes to autoresearch can take control for cost-effective research to occur.

	▲	stingraycharles 5 hours ago \| parent [-]
		Yes but at that point you may as well use a proper hyperparameter tuning framework like optuna if all the LLM agent is supposed to do is do hyperparameter tuning.

▲

mandevil 15 hours ago | parent | prev [-]

Optuna or skopt are open source and won't take any GPU time at all to do it.

	▲	janalsncm 13 hours ago \| parent [-]
		Optuna requires exploring the hyperparameter space which means running the experiments with those hyperparameters. For a fixed search space it will almost certainly be better though.