I have been the first (and only) DevOps person at a couple startups. I'm usually pretty guilty of NIH and wanting to develop in-house tooling to improve productivity. But more and more in my career I try to make boring choices.

Cost is usually not a huge problem beyond seed stage. Series A-B the biggest problem is growing the customer base so the fixed infra costs become a rounding error. We've built the product and we're usually focused on customer enablement and technical wins - proving that the product works 100% of the time to large enterprises so we can close deals. We can't afford weird flakiness in the middle of a POC.

Another factor I rarely see discussed is bus factor. I've been in the industry for over a decade, and I like to be able to go on vacation. It's nice to hand off the pager sometimes. Using established technologies makes it possible to delegate responsibility to the rest of the team, instead of me owning a little rats nest fiefdom of my own design.

The fact is that if 5k/month infra cost for a core part of the service sinks your VC backed startup, you've got bigger problems. Investors gave you a big pile of money to go and get customers _now_. An extra month of runway isn't going to save you.

▲

woooooo 5 hours ago | parent [-]

The issue is when all the spending gets you is more complexity, maintenance, and you don't even get a performance benefit.

I once interviewed with a company that did some machine learning stuff, this was a while back when that typically meant "1 layer of weights from a regression we run overnight every night". The company asked how I had solved the complex problem of getting the weights to inference servers. I said we had a 30 line shell script that ssh'd them over and then mv'd them into place. Meanwhile the application reopened the file every so often. Zero problems with it ever. They thought I was a caveman.

▲

ffsm8 4 hours ago | parent [-]

The issue with solutions like that is usually that people don't know how it works and how to find it if it ever stops working...

Basically discoverability is where shell script fail

▲

roncesvalles 19 minutes ago | parent | next [-]

You can literally have a 20 line Python script on cron that verifies if everything ran properly and fires off a PagerDuty if it didn't. And it looks like PagerDuty even supports heartbeat so that means even if your Python script failed, you could get alerted.

▲

chuckadams 4 hours ago | parent | prev | next [-]

Those scripts have logs, right? Log a hostname and path when they run. If no one thinks to look at logs, then there's a bigger problem going on than a one-off script.

▲

LevGoldstein 2 hours ago | parent | prev | next [-]

Which is why you take the time to put usage docs in the repo README, make sure the script is packaged and deployed via the same methods that the rest of the company uses, and ensure that it logs success/failure conditions. That's been pretty standard at every organization I've been at my entire professional career. Anyone who can't manage that is going to create worse problems when designing/building/maintaining a more complex system.

	▲	mlyle 8 minutes ago \| parent [-]
		Yah. A lot of the complexity in data movement or processing is unneeded. But decent standardized orchestration, documentation, and change management isn't optional even for the 20 line shell script. Thankfully, that stuff is a lot easier for the 20 line standard shell script. Or python. The python3 standard library is pretty capable, and it's ubiquitous. You can do a lot in 50-100 lines (counting documentation) with no dependencies. In turn it's easy to plug into the other stuff.

▲

woooooo 4 hours ago | parent | prev | next [-]

That becomes a problem if you let the shell script mutate into an "everything" script that's solving tons of business problems. Or if you're reinventing kubernetes with shell scripts. There's still a place for simple solutions to simple problems.

▲

justsomehnguy 4 hours ago | parent | prev [-]

> Basically discoverability is where shell script fail

No, it's lack of documentation and no amount of $$$$/m enterprise AI solutions (R)(TM) would help you if there is no documentation.