| ▲ | ghm2180 2 hours ago | |
I suspect it's regressed because they actually don't have good e2e evals comparing the two experiences to test the regression and UX. Like the simplest evals you want is just a shit ton of STTs across human speech accounting for dialects, accents etc and then figuring out how the distribution of errors has moved or increased on the axis you care about. That they likely have. But then you have to integrate that into the product to capture the entire funnel of errors, which I suspect is the part they don't have. The stacks are not "vertically integrated" to test quickly. | ||