This is really cool! Is there an alternative way of thinking about it involving a hidden markov model, looking for a change in value of an unknown latent P(fail)? Or does your approach end up being similar to whatever the appropriate Bayesian approach to the HMM would be?

The HMM framing connects to change-point detection. CUSUM (Cumulative Sum) charts solve a related problem: detecting when a process parameter has shifted by accumulating deviations from an expected value.

Key difference: CUSUM assumes sequential observation and asks "when did the distribution shift?" Bayesect asks "which commit should I test next?" — active learning vs passive monitoring.

But they could complement each other. If you already have CI pass/fail history, CUSUM on that data gives you a rough change-point estimate for free (no extra test runs), then bayesect refines it with active sampling.