I started building https://github.com/kcc999/sidechain
It's a CLI Framework for running evaluations against LLMs.