> This was costing us ~$300K/year in compute, and the number kept growing as more customers and detection rules were added.

Maybe I’m out of touch, but I cannot fathom this level of cost for custom lambda functions operating on JSON objects.

▲

otterley 9 hours ago | parent | next [-]

They said in the article that they were running up to 200 pods at a time. Doing some back of the envelope math, 200 pods at $300,000 year is about $0.17/hour, which is exactly what an EC2 c5.xlarge costs per hour (on demand). That has 4 vCPUs, so about 800 vCPUs during peak, with $0.0425/CPU-hour.

I do have some questions like:

* Did they estimate cost savings based on peak capacity, as though it were running 24x7x365?

* Did they use auto scaling to keep costs low?

* Were they wasting capacity by running a single-threaded app (Node-based) on multi-CPU hardware? (My guess is no, but anything is possible)

	▲	ebb_earl_co 9 hours ago \| parent [-]
		This is a helpful breakdown, thanks, @otterley. It is, by orders of magnitude, larger than any deployment that I have been a part of in my work experience, as a 10-year data scientist/Python developer.

▲

jcims 9 hours ago | parent | prev | next [-]

This is where the cost came from.

>The reference implementation is JavaScript, whereas our pipeline is in Go. So for years we’ve been running a fleet of jsonata-js pods on Kubernetes - Node.js processes that our Go services call over RPC. That meant that for every event (and expression) we had to serialize, send over the network, evaluate, serialize the result, and finally send it back.

But either way, we're talking $25k/mo. That's not even remotely difficult to believe.

▲

manquer 9 hours ago | parent | prev | next [-]

First I thought they were AWS lambda functions, perhaps possible if they are over-provisioned for very concurrency or something similar $25k/month is in realm of possibility.

But no, the the post is talking about just RPC calls on k8s pods running docker images, for saving $300k/year, their compute bill should be well above $100M/year.

Perhaps if it was Google scale of events for billions of users daily, paired with the poorest/inefficient processing engine, using zero caching layer and very badly written rules, maybe it is possible.

Feels like it is just an SEO article designed to catch reader's attention.

▲

slopinthebag 10 hours ago | parent | prev [-]

It has to be satire right? Like, you aren't out of touch on this. I get engineers maybe making the argument that $300k / year on cloud is the same as 1.5 devops engineers managing in-house solutions, but for just json parsing????

▲

xp84 9 hours ago | parent | next [-]

For numbers like that, I can never tell whether it's just a vastly larger-scale dataset than any that I've seen as a non-FAANG engineer, OR, a hilariously-wasteful application of "mAnAgEd cLoUd sErViCeS" to a job that I could do on a $200/month EC2 instance with one sinatra app running per core. This is a made-up comparison of course, not a specific claim. But I've definitely run little $40 k8s clusters that replaced $800/month paid services and never even hit 60% CPU.

	▲	ebb_earl_co 9 hours ago \| parent [-]
		Right, this is roughly my mental situation, too. I guess that streaming JSON things can eat up compute way faster than I had any intuition for!

▲

encoderer 10 hours ago | parent | prev [-]

I wonder if you've ever worked on a web service at scale. JSON serialization and deserialization is notoriously expensive.

▲

bawolff 9 hours ago | parent | next [-]

They got a 1000x speed up just by switching languages.

I highly doubt the issue was serialization latency, unless they were doing something stupid like reserializing the same payload over and over again.

▲

encoderer 9 hours ago | parent [-]

Well, for starters, they replace the RPC call with an in-process function call. But my point is anybody who's surprised that working with JSON at scale is expensive (because hey it's just JSON!) shouldn't be surprised.

	▲	bawolff 9 hours ago \| parent [-]
		Well everything is expensive at scale, and any deserialization/serialization step is going to be expensive if you do it enough. However yes i would be surprised. JSON parsing is pretty optimized now, i suspect most "json parsing at scale is expensive" is really the fault of other parts of the stack

▲

leptons 9 hours ago | parent | prev | next [-]

It can be, but $500k/year is absurd. It's like they went from the most inefficient system possible to create, to a regular normal system that an average programmer could manage.

I have no idea if they are doing orders of magnitude more processing, but I crunch through 60GB of JSON data in about 3000 files regularly on my local 20-thread machine using nodejs workers to do deep and sometimes complicated queries and data manipulation. It's not exactly lightning fast, but it's free and it crunches through any task in about 3 or 4 minutes or less.

The main cost is downloading the compressed files from S3, but if I really wanted to I could process it all in AWS. It also could go much faster on better hardware. If I have a really big task I want done quickly, I can start up dozens or hundreds of EC2 instances to run the task, and it would take practically no time at all... seconds. Still has to be cheaper than what they were doing.

▲

slopinthebag 9 hours ago | parent | prev [-]

Would it be better or worse if I had that experience and still said it's stupid?

▲

encoderer 9 hours ago | parent [-]

You didn't say it was stupid. If you had, I would have just ignored the comment. But you expressed a level of surprised that led me to believe you're unfamiliar with how much of a pain in the ass JSON parsing is.

	▲	slopinthebag 9 hours ago \| parent [-]
		I think OP’s point was surprise that a company would spend so much on such inefficient json parsing. I’m agreeing. I get that JSON is not the fastest format to parse, but the overarching point is that you would expect changes to be made well before you’re spending $300k on it. Or in a slightly more ideal world, you wouldn't architect something so inefficient in the first place. But it's common for engineers to blow insane amounts of money unnecessarily on inefficient solutions for "reasons". Sort of reminds me of saas's offering 100 concurrent "serverless" WS connections for like $50 / month - some devs buy into this nonsense.