Can someone shed some lights on what temporal does? Their blog says [1]

Durable Execution offers three key benefits:

    It improves application reliability by providing fault tolerance.
    It simplifies code by allowing it to focus on the goal instead of potential problems.
    It accelerates development by eliminating the need to write complex error-handling logic.

Why is it such a big deal? Genuine question, not trying to be snarky

[1] https://temporal.io/blog/what-is-durable-execution

It’s an async job runner that scales well. Yeah they advertise a lot of fancy features, but really what they are selling is solid failure-resistant infra. Companies could set the same thing up with an in-house team, but it’s pretty great to be able to click a button and start running workloads at tens or hundreds of thousands of RPS without needing to figure out the internals.

Say you've got a process that spans a bunch of services and touches the real world. Some ecommerce check out for example.

You need to reserve some inventory, charge a card, eventually email a customer and steps can go wrong. So you have queues, and retries and ways to back things out, undo changes.

To my understanding, Temporal's idea is to factor that part out, the queues and retries, and offsetting actions, so you write the logic and not the workflow orchestration.

I think it's useful for implementing Sagas https://temporal.io/blog/saga-pattern-made-easy

Basically, a multi-step event handling system, where the data could be spread across multiple databases/systems, so there's no way to rollback a transaction atomically across all the databases. Instead, you explicitly codify "compensations" to undo your previous commits so that you eventually end up in a consistent state.

Temporal is the orchestrator/framework/library to implement the above in an easier way.

In concrete terms if your app has functions that do very little work but run for hours or days at a time blocking on external resources, and it is acceptable to add 100+ms latency + a serialization boundary to every function call, it is acceptable to take on a bunch of other tooling complexity, and you are willing to sacrifice verifiability of the code, then it might be a fit. In those situations the structure of it (or DBOS or similar) can help standardize that part of your app. Unlike a regular state machine based approach it's easier to represent branching logic and so forth in imperative code. I think its popularity is like the Mongo NoSQL fad, it looks convenient up front but in practice is very painful in most situations. Having used it a fair amount I would recommend starting with actors and state machines, especially in this age of robot code where it's easier to apply heavier approaches to testing in verification.

Asking what temporal is? The founders (Maxim Fateev and Samar Abbas) originally made Amazon Simple Workflow Service (SWF), Azure Durable Task Framework (DTF), and Uber Cadence. Temporal is the latest in the long line of workflow systems they've built. It's a task engine with state, recovery and highly scalable. It's cool stuff, and it simplifies many business workflows when it comes to recovery and scaling. There are many similar systems on the market today that mimic durable functions but Temporal is arguably the best.

When you write traditional distributed systems/cloud native code you have to handle all the failures and retries somewhere. Anyone who's run an Ansible playbook or similar and had a failure halfway through leave the state of the systems in a weird half-state is familiar. Or dealt with spot preemptions or nodes failing or getting unlucky with OOM killer or hardware failures or a myriad of other failure modes.

In Temporal, you use their SDK to mark which code is either:

- Deterministic without external dependencies on network, disk, clock, etc.

- Non-deterministic (e.g. accesses a filesystem, dependent on clock time, talks over the network)

You can write the code without handling flakiness or retries and the Temporal control plane handles all the progress tracking and retries for you. Progress is tracked at the individual line of code for deterministic code, or for non-deterministic code, tracked at function boundaries defined by the programmer. You buy into more complexity upfront, but it makes the application code way simpler and easier to manage overall.

They do a lot of other cool stuff on top of all this, and their Temporal Worker Controller architecture is particularly well suited to running massive scale processing/AI workloads on Kubernetes (handles a lot of stuff like autoscaling without interrupting work, rainbow version rollout, etc.)

temporal makes you split up your async tasks explicitly into a message (a request to fulfill a task, thrown up to a temporal server), and a task handler (pulling a tasks-request from a queue, from the temporal server), rather than both wrapped in a vanilla request/response cycle.

temporal centralizes all the error handling, retry handling, ... some web pages to manage failed, pending tasks.

it's been good for us, but a real step function in complexity of the app.

Thank you for all the explanations. I think I have better understanding now. Another question, at what scale (in terms of number of services, and maybe throughput? what other dimensions?) should we start considering using temporal?

This is a rough explanation, but generally it does a couple of things that are helpful.

Functionally, it's a "workflow" runner (ex - you can mostly treat it like a queue, where you've got workers that are picking up work to do).

But it wraps a couple of pretty handy features on top like:

- It preserves most arguments to actions, and it makes system details deterministic for retries (ex you can re-run a workflow at a later time, and temporal will make sure the code sees details like date and time as though it were the original run, and will yell at you if you try to write code that won't be deterministic on retries)

- It supports very long waits easily. (ex - very easy to have a workflow do a couple things, wait a week, then do some more things).

- It has decent profiling and UI tools

- It can "restart" a failed workflow deterministically from the step at which it failed (using details from the original point)

---

Basically - it's a background worker service that's put a lot of time and thought into ways to handle failures better.

It absolutely still has some considerable pain points though, and I find it difficult to use for larger tasks (ex - their message gRPC size limit of 4mb is a b*&^% to work around, since it often breaks a lot of the utility they provide, and the history cap at 50mb is also really painful in certain situations.)

Really - I think it was just the right tool at the right time to make calling LLMs with long waits relatively easy and somewhat foolproof.

"Why is it such a big deal?"

It's used by OpenAI.

> So... like exception handling?

Yes...with eventual consistency.

If your process OOMs, if the machine restarts, etc.

Plus some other distributed execution advantages.