The issue is when all the spending gets you is more complexity, maintenance, and you don't even get a performance benefit.

I once interviewed with a company that did some machine learning stuff, this was a while back when that typically meant "1 layer of weights from a regression we run overnight every night". The company asked how I had solved the complex problem of getting the weights to inference servers. I said we had a 30 line shell script that ssh'd them over and then mv'd them into place. Meanwhile the application reopened the file every so often. Zero problems with it ever. They thought I was a caveman.

▲

ffsm8 4 hours ago | parent [-]

The issue with solutions like that is usually that people don't know how it works and how to find it if it ever stops working...

Basically discoverability is where shell script fail

▲

roncesvalles 16 minutes ago | parent | next [-]

You can literally have a 20 line Python script on cron that verifies if everything ran properly and fires off a PagerDuty if it didn't. And it looks like PagerDuty even supports heartbeat so that means even if your Python script failed, you could get alerted.

▲

chuckadams 4 hours ago | parent | prev | next [-]

Those scripts have logs, right? Log a hostname and path when they run. If no one thinks to look at logs, then there's a bigger problem going on than a one-off script.

▲

LevGoldstein an hour ago | parent | prev | next [-]

Which is why you take the time to put usage docs in the repo README, make sure the script is packaged and deployed via the same methods that the rest of the company uses, and ensure that it logs success/failure conditions. That's been pretty standard at every organization I've been at my entire professional career. Anyone who can't manage that is going to create worse problems when designing/building/maintaining a more complex system.

	▲	mlyle 5 minutes ago \| parent [-]
		Yah. A lot of the complexity in data movement or processing is unneeded. But decent standardized orchestration, documentation, and change management isn't optional even for the 20 line shell script. Thankfully, that stuff is a lot easier for the 20 line standard shell script. Or python. The python3 standard library is pretty capable, and it's ubiquitous. You can do a lot in 50-100 lines (counting documentation) with no dependencies. In turn it's easy to plug into the other stuff.

▲

woooooo 4 hours ago | parent | prev | next [-]

That becomes a problem if you let the shell script mutate into an "everything" script that's solving tons of business problems. Or if you're reinventing kubernetes with shell scripts. There's still a place for simple solutions to simple problems.

▲

justsomehnguy 4 hours ago | parent | prev [-]

> Basically discoverability is where shell script fail

No, it's lack of documentation and no amount of $$$$/m enterprise AI solutions (R)(TM) would help you if there is no documentation.