Hasn't there been a lot talk about Astra's opaque reasoning capabilities (being able to think through complex questions without using a chain of thought)?

Given that, can't you just replicate Jev by telling Astra "here is the question, you must make a multiple choice decision / output a score between 1-10, please answer directly in a single word, no reasoning allowed"?

(Edit: Ok, Jev is much cheaper in input tokens so these two aren't directly comparable at all)

the edit is right - jev would be cheaper, faster, and more self-consistent (in general)

we actually use astra (and fable) in this way for our evals: evals.typesafe.ai

someone on the team cooked hard on that and it shows example traces comparing our model to opus/sol