Awesome work, but I am genuinely curios about the use cases where the 200ms init time being a problem?

nemothekid 3 days ago | parent | next [-]

I have a lambda that runs inference on a model for a feature that is seldomly used.

Lambda is far more economic given the memory requirements, but we recently moved to Rust+Candle to shave off ~300ms on cold starts as the lag could be really jarring.

▲

ericmcer 3 days ago | parent | prev | next [-]

Serverless implementations where lambdas are being used for APIs so +200ms on a single call is not good.

▲

bastawhiz 3 days ago | parent [-]

It's not 200ms per call. It's 200ms cold start. Once a lambda is started, you don't pay that cost again. In most cases, you pay this a few times after a deployment or when traffic is increasing and not again.

	▲	hvb2 3 days ago \| parent [-]
		> you pay this a few times after a deployment or when traffic is increasing and not again. This is now how lambda works. A lambda instance sticks around for 10s of minutes at most. So you'll have cold starts every day no matter how often you deploy. If you have very little traffic you might actually have cold starts often as the lifetime of a lambda is determined based on its use. Unfortunately AWS has always been rather unspecific about this behavior but here's a link to a pluralsight blog https://www.pluralsight.com/resources/blog/cloud/how-long-do...

▲

ttoinou 3 days ago | parent | prev [-]

Maybe its a problem when you’re paying 200ms per small request and you have millions of them ?

	▲	hnlmorg 3 days ago \| parent \| next [-]
		The hypothetical 200ms is for cold starts. That means when the docker container that runs behind the lambda first starts up. When a lambda is invocated, that docker container isn’t immediately destroyed. It’s kept around for reuse. So there isn’t a realistic scenario where you would get a million cold starts in a month.
	▲	Eric_WVGG 3 days ago \| parent \| prev \| next [-]
		Some companies who actually depend on actions like this should consider fronting some cash to keep this project going.
	▲	bastawhiz 3 days ago \| parent \| prev [-]
		If you have cold starts on all of your millions of requests, something is very wrong with your code.