> This is a bit related to external dependencies vs build yourself (AKA reinvent the wheel). Quite often the external library, long term, causes more issues than building it yourself (assuming you _can_ build a competent implementation).

I feel like this happens mostly because simpler is better, and most of these dependencies don’t follow a good “UNIX” philosophy of modularity, being generic etc. something that you’d notice the standard libraries try to achieve.

Most of these third party dependencies are just a very specific feature that starts to add more use cases until it becomes bloated to support multiple users with slightly different needs.

> Want to write ML, probably best to use a language with functions as first class types (ie a FP language). You might say, most people doing ML use Python. This came to be because the language was picked based upon what people knew. But the big companies doing ML (successfully) don't use Python anymore and haven't for over a decade. ML researchers kept FP alive for several decades when nobody else cared because FP is the best tool for ML (or for writing a compiler).

I strongly disagree. The top AI companies are using a lot of Python (although sure they also use other languages too, but they're definitely using Python!). Even if by ML you mean old-school ML techniques, a ton of big companies also use Python for this (some might use MATLAB or R).

On the other hand I don't know of a single large company using an FP language for ML, unless you count something like Spark (which I would push back on: the Scala API of Spark is not really FP and almost all users of Spark that I know of program mainly in the more OO part of Scala rather than its FP part).

Even die-hard FP companies such as Jane Street use Python for their ML (see e.g. https://www.janestreet.com/join-jane-street/position/4276720... which notably mentions Python and does not mention OCaml).

Do you know of any company with a team of over 50 ML researchers (either old-school ML or modern AI) using an FP language as their primary workhorse for that team? Because I can't think of a single one.

More to the point, do you know of any ML researcher (and who is acknowledged as primarily an ML researcher by other ML researchers) who primarily programs in an FP language?

Even in the golden era of symbolic AI from the 70s and 80s they still weren't using FP languages (they basically didn't exist yet). The closest you could say is that they were using Lisps, but Lisps aren't FP languages by default. Some are (e.g. Clojure), but many aren't or at least aren't any more FP than any other multiparadigmatic language (e.g. Common Lisp). And again I don't know of any significant ML work being done in Lisps at this point (there's some scattered small teams and individuals doing work there, but nothing that I think could rise to the level of "big ML company").

I think for many cases it is fair to make the assumption that you're staying roughly within the same kind of application. So if your team previously did web applications you won't suddenly do embedded stuff or high-frequency trading. And within such an area, there aren't enough differences between languages to overcome the drawback of learning a new one.

I think you missed something...

> ... or there are underlying reasons for a language shift.

As to "best language" that is just as dogmatic as anything else... just look at the C/C++ vs. Rust divide in the Linux community.

I think you are overestimating the value of a best fit language for any given task, especially those where there are a half dozen popular languages that more people know well that can do the job good enough. Don't build for a sky scraper when all you need is a birdhouse.

Also, MOST engineers aren't particularly talented. If you're fortunate enough to be working for an organization where everyone is a rockstar, that's great... for those doing bog standard CRUD apps for business, you don't get rockstar money, and you aren't finding rockstar talent. You get what you get and make the pest of it.

In nearly three decades, I've once, only once worked a project where I didn't have to explain a relatively simple concept to someone, where everyone on the project delivers their pieces in time and all were talented. It was wonderful. Then new management gets stacked on top, all the job roles are reclassified to mid level developers and everyone rolls out of that group.

A lot of the actual experience is literally explaining public/private key usage to other developers who manage to (re)use the same keys from dev to all the production deployments. Or a pissing match with the "security expert" who doesn't understand that your app's use case is different than the in the box security script that is failing, because your /login route is a different app from / and the bogus query params don't matter.