▲ | chambers 11 hours ago | |
Statsig's core value is their experimentation platform— the automation of Data Science. Big Tech teams want to ship features fast, but measuring impact is messy. It usually requires experiments and traditionally every experiment needed one Data Scientist (DS) to ensure statistical validity, i.e., "can we trust these numbers?". Ensuring validity means DS has to perform multiple repetitive but specialized tasks throughout the experiment process: debugging bad experiment setups, navigating legacy infra, generating & emailing graphs, compensating for errors and biases in post-analysis, etc. It's a slog for folks involved. Even then, cases still arise where Team A reports wonderful results & ships their feature while unknowingly tanking Team B's revenue— a situation discovered only months later when a DS is tasked to trace the cause. Experimentation platforms like Statsig exist to lower the high cost of experimenting. To show a feature's potential impact before shipping, while reducing frustrations along the way. Most platforms will eliminate common statistical errors or issues at each stage of the experiment process, with appropriate controls for each user role. Engs setup experiments via SDK/UI with nudges and warnings for misconfigurations. DS can focus on higher-value work like metric design. PMs view shared dashboards and get automatic coordination emails with other teams if their feature is seen as breaking. People still fight but earlier on and in the same "room" with fewer questions about what's real versus what's noise. Separating real results from random noise is the meaning of "statsig" / "statistically significant". I think it's similar to how companies define their own metrics (their sense of reality) while the platform manages the underlying statistical and data complexity. The ideal outcome is less DS needed, less crufty tooling to work around, less statistics learning, and crucially, more trust & shared oversight. But it comes at considerable, unsaid cost as well. Is Statsig worth $1B to OpenAI? Maybe. There's an art & science to product development, and Facebook's experimentation platform was central to their science. But it could be premature. I personally think experimentation as an ideology best fits optimization spaces that previously achieved strong product-market fit ages ago. However, it's been years since I've worked in the "Experimentation" domain. I've glossed over a few key details in my answer and anyone is welcome to correct me. | ||
▲ | siva7 10 hours ago | parent [-] | |
If such platforms are the result of what facebook is today, it's not exactly an advertisement for these products. |