Fun memories.

We have successfully replaced thousands of complicated deep net time series based anomaly detectors at a FANG with statistical (nonparametric, semiparametric) process control ones.

They use 3 to 4 orders lower number of trained parameters and have just enough complexity that a team of 3 or four can handle several thousands of such streams.

The amount of baby sitting that deep net models needed was astronomical, debugging and understanding what has happened quite opaque.

For small teams, with limited resources I would still heavily recommend stats based models for time series anomaly detection.

May not be your best career move right now for political reasons. Those making massive bets do not like to confront that some of their bets might not have been well placed. They may try to make it difficult for contrary evidence to become too visible.

▲

auxiliarymoose 4 hours ago | parent | next [-]

Super cool, thanks for sharing!

This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.

You'd think that rational businesses would take the low-risk snooze-fest high-margin option any day instead of unintelligible and unreliable options that demand a lot of resources, and yet...

▲

DavidPeiffer 3 hours ago | parent | next [-]

>This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.

In 2013 my statistics professor warned that once we are in the real world, "people will come up to you trying to sell fancy machine learning models for big money, though the simple truth is that many problems can be solved better by applying straightforward statistical methods".

There has always been the ML hype, but the last couple years are a whole different level.

▲

srean 3 hours ago | parent | prev | next [-]

It does not work that way in the short term.

Say you have bet billions as a CEO, CTO, CFO. The decision has already been made. Such a steep price had to come at the cost of many groups and teams and projects in the company.

Now is not a time to water plants that offer alternatives. You will have a smoother ride choosing tools that justifies that billion dollar bet.

▲

auxiliarymoose 3 hours ago | parent [-]

Decision-making in organizations is definitely a hard problem.

I think an uncomfortable reality is that a lot of decisions (technology, strategy, etc.) are not optimal or even rational, but more just an outcome of personal preferences.

Even data-driven approaches aren't immune since they depend on the analysis and interpretation of the data (which is subjective).

▲

srean 3 hours ago | parent [-]

Data informed is good. Purely data driven is a bad idea.

After all even in Physics big advances came from thought experiments. Data is one way to reason about a decision, logic and knowledgebase is another way. Both can be very powerful if one retains the humility of fallibility.

In organizations one common failure mode is that the organisational level at which decisions are made are not the same levels where the decisions are going to have their effects felt.

It's a really difficult problem to solve. Too much decentralisation is also a bad idea. You get the mess of unplanned congested cities.

	▲	cb321 2 hours ago \| parent [-]
		For a while now, I've been summarizing the ease with which everything turns into a "Humanity Complete" problem via: "Delegation affords so much, but trust sure is tricky." This has been observed forever in various forms/contexts. Planning & policy people call them "Wicked Problems" (https://en.wikipedia.org/wiki/Wicked_problem). The Philosophy of Science one goes by the Demarcation Problem (https://en.wikipedia.org/wiki/Demarcation_problem) { roughly, in the sense that the really hard nugget connects to "trust" }. At least one aspect of all of it is that trust is a little like money/capital and "faking it" is a bit like "stealing". The game theory of it is that since faking is virtually always vastly cheaper there are (eventually) huge incentives to do so, at some point by someone(s). So, almost any kind of trust/delegation structure has a strong pull toward "decay", from knock-off brands to whatever. It just takes a sadly small fraction of Prisoner's Dilemma defectors to ruin things/systems thereof. 2nd law of thermo makes order cost energy and this decay feels like almost an isomorphic (maybe even the same..?) thing. It's not just product/tech enshittification, but that might be yet another special case/example. Anyway, I have no great answers or as some responder to me a while back said, if I did, I'd "have a Nobel and possibly be the first president of the united planet".

▲

menaerus 4 hours ago | parent | prev [-]

> There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.

I know some examples but not too many. Care to share more examples?

▲

auxiliarymoose 3 hours ago | parent | next [-]

Some off the top of my head...

- Instead of trying to get LLMs to answer user questions, write better FAQs informed by reviewing tickets submitted by customers

- Instead of RAG for anything involving business data, have some DBA write a bunch of reports that answer specific business questions

- Instead of putting some copilot chat into tools and telling users to ask it to e.g. "explain recent sales trends", make task-focused wizards and visualizations so users can answer these with hard numbers

- Instead of generating code with LLMs, write more expressive frameworks and libraries that don't require so much plumbing and boilerplate

Of course, maybe there is something I am missing, but these are just my personal observations!

▲

menaerus 3 hours ago | parent [-]

With all due respect, all of those examples are the examples of "yesterday" ... that's how we have been bringing money to businesses for decades, no? Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned.

Businesses are incentivized to be more productive and cost-effective since they are solely profit-driven so they naturally see this as an opportunity to make more money by hiring less people while keeping the amount of work done roughly the same or even more.

So "classical" approach to many of the problems is I think the thing of a past already.

	▲	auxiliarymoose 3 hours ago \| parent [-]
		> Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned. We really don't. There are demos that look cool onstage, but there is a big difference between "in store good" and "at home good" in the sense that products aren't living up to their marketing during actual use. IMO there is a lot of room to grow within the traditional approaches of "yesterday" - The problem is that large orgs get bogged down in legacy + bureaucracy, and most startups don't understand the business problems well enough to make a better solution. And I don't think that there is any technical silver bullet that can solve either of these problems (AI or otherwise)

▲

srean 3 hours ago | parent | prev [-]

In the realm of data science, Linear models and SAT solvers used cleverly will get you a surprisingly long way.

▲

menaerus 3 hours ago | parent | next [-]

I thought the OCR was one of the obvious examples where we have a classical technology that is already working very well but in the long-run I don't see it surviving. _Generic_ AI models already can do the OCR kinda good but they are not even trained for that purpose, it's almost incidental - they've never been trained to extract the, let's say name/surname from some sort of a document with a completely unfamiliar structure, but the crazy thing is that it does work somehow! I think that once somebody finetunes the AI model only for this purpose I think there's a good chance it will outperform classical approach in terms of precision and scalability.

▲

srean 3 hours ago | parent [-]

In general I agree. For OCR I agree vehemently. Part of the reason is the structure of the solution (convolutions) match the space so well.

The failure cases are those where AI solutions have to stay in a continuous debug, train, update mode. Then you have to think about the resources you need, both in terms of people as well as compute to maintain such a solution.

Because of the way the world works, it's endemic nonstationarity, the debug-retrain-update is a common state of affairs even in traditional stats and ML.

▲

menaerus 2 hours ago | parent [-]

I see. Let's take another example here, I hope I understood you - imagine you have an AI model which is connected to all of your company's in-house data generation sources such as wiki, chat, jira, emails, merge requests, excel sheets, etc. Basically everything that can be deemed useful to query or to create business inteligence on top of. These data sources are continously generating more and more data every day, and given their nature they are more or less unstructured.

Yet, we have such systems in place where we don't have to retrain the model with ever-growing data. This is one example I could think of but it kinda suggests that models, at least for some purposes, don't have to be retrained continuously to keep them running well.

I also use a technique of explaining something to the AI model he has not seen before (according to the wrong answer I got from it previously), and it manages to evolve the steps, whatever they are, so that it gives me the correct answer in the end. This also suggests that capacity of the models is larger than what they have been trained on.

	▲	srean 2 hours ago \| parent [-]
		Data science solutions are different in the sense they rarely ever get done and dusted in a sense a sorting library might. There's almost always something or the other breaking. Did the nature of data change. Did my upstream data feed change. Why are these small set of examples not working for this high paying customer. You would need resources to understand and fix these problems quarter after quarter. A rich network of data dependencies can be a double edged sword. Rarely are upstream code and data changes benign to the output of the layer you own. There are two cases where AI solutions are perfect. They are so good that they are fire and forget. The second is that your customer is a farmer not a gardener. Individual failing saplings mean little to him. If a single misbehaving plant can cause commercially significant damage then when choosing opaque tools you must consider the maintenance cost you may be signing up for. Say I have a ton of historical data that is being continuously added to. It's a real temptation to replace the raw data with a model that uses less number of parameters than the raw data. In a sense lossy compression. Can be a very bad idea. Data instances where the model does not fit well may be the most important pieces of art information. Model paints with a broad brush stroke. If you are hunting faults, you have been aware that a lossy compression can paper them away. You are also potentially harming a future model that could have been trained but you have thrown away a decade of useful data because storage costs were running so high. No easy solution. General recommendation would be to compress but losslessly simply because you know not what may be valuable in the future. If it's impossible, then so be it, you have to eat that opportunity cost in the future, but you did your best.

▲

jmalicki an hour ago | parent | prev [-]

I've seen a lot of uses for SAT solvers, but what do you use them for in data science? I can't find many references to people using them in that context.

	▲	srean an hour ago \| parent [-]
		Root causing from symptoms is one case where SAT or their ML analogue -- graphical models are quite useful.

▲

esafak 30 minutes ago | parent | prev | next [-]

Can you be more specific about what SPC algorithm you moved to? Did you trade off prediction quality for complexity, increasing the number of false alarms?

▲

traceroute66 an hour ago | parent | prev | next [-]

> We have successfully replaced thousands of complicated deep net time series based anomaly detectors at a FANG with statistical (nonparametric, semiparametric) process control ones.

Interesting.

Were you using things like Matrix Profile too ? And if so, have those been replaced too ?

	▲	srean an hour ago \| parent [-]
		In this specific project no, but in others a very emphatic yes.

▲

kqr 2 hours ago | parent | prev | next [-]

What confuses me about deep nets is that there's rarely enough signal to be able to meaningfully train a large number of parameters. Surely 99 % of those parameters are either (a) incredibly unstable, or (b) correlate perfectly with other parameters?

▲

srean 2 hours ago | parent [-]

They do. There are enormous redundancies. There's a manifold over which the parameters can vary wildly yet do zilch to the output. The nonlinear analogue of a null space.

Parameter instability does not worry a machine learner as much as it worries a statistician. ML folks worry about output instabilities.

The current understanding goes that this overparameterization makes reaching good configurations easier while keeping the search algorithm as simple as stochastic gradient descent.

▲

kqr 2 hours ago | parent [-]

Huh, I didn't know that! Are there efforts to automatically reduce the number of parameters once the model is trained? Or do the relationships between parameters end up too complicated to do that? I would assume such a reduction would be useful for explainability.

(Asking specifically about time series models and such.)

	▲	srean an hour ago \| parent [-]
		What you are looking for is the lottery ticket hypothesis for neural networks. Hit a search engine with those words you will find examples. https://arxiv.org/abs/1803.03635 ( you can follow up on semantic scholar for more) Selecting which weights to discard seems as hard as the original problem. But random decimation, sometimes barely informed decimation have been observed to be effective. On the theory side now it's understood that in the thicket of weights, lurk a much much smaller subset that can have nearly the same output. These observations are for DNNs in general. For time series specifically I don't know what the state of the art is. In general NNs are still catching up with traditional stats approaches in this domain. There are a few examples where traditional approaches have been beaten, but only a few. One good source to watch are the M series of competitions.

▲

oulipo2 2 hours ago | parent | prev [-]

could you give a brief overview of: - what libs were you using - what kind of algos / models were most useful for what kind of data?

I have an IoT use-case, I wanted to look both at NNs and more classical stats models to see if it has value

	▲	srean 2 hours ago \| parent [-]
		Can't for obvious reasons. But no specialized libraries used. The usual Python stack that comes packaged for any respectable OS distribution these days, mixed in with other close-to-the-metal languages for performance or API compatibility reasons. Look up nonparametric statistical process control and you will find useful papers. The algorithms are actually quite simple to implement. If the algorithms are not simple then probably they are not worth your time. The analysis in the paper might be complicated, don't worry about that, look for simplicity of the algorithms.