Exactly the same case here. A few years ago, pre-ChatGPT, my wife was working with a massive database in SPSS or R (can't remember which). Now these "databases" are really just huge CSV/TSV files, with millions of records, one record per line. She was applying some very complicated mathematical functions to every single row, and she was frustrated because it was going to take days to apply a particular calculation she was working on to the whole database.

So I took a look for about 30 mins figuring out the code (I don't know SPSS or R), and it turns out she was doing some calculations that were identical for every single line, regardless of that line's fields. So I pre-calculated a few things, prior to starting the per-record-loop, and then just used the pre-calculated values every time. After that, every run took less than 5 minutes. She's amazing at math and economics, but optimizing low-level code is not something she has done in the past.

Yeah, very similar. For each and every image, the code loaded every image into memory, several times each, then did a pixel-by-pixel walk through the entire image, identified all objects, and then compared all of the objects to all of the objects from all of the previous images (which it would not use the calculated result for, but rather recompute that too) to establish which one was moving - or which one was staying still.

The main fix was not making it hideously IO bound, as not only was it loading every image multiple times, it was then grinding away in swap as we’re talking tens of gigabytes of raw image data, in an era when 1gb of ram was a lot - and then making it spiral out from the last known location of the object of interest rather than brute-forcing it. My solution wasn’t even optimal, as it was literally just an afternoon of tooling around as a favour.