We are working on a workflow engine as well - picking up concepts from the past - not quite as far back as mainframes, but I respect looking back to the past for things that work well, and re-applying them to a new world. I'd be up for sharing notes. Our work is here https://github.com/sartography/spiff-arena -- live demo is here https://spiff.works/agent-demo

Hi, thank you for your feedback.

I think in the past, I've already had a look at your repo, but it's a little bit different from my idea. Your is build around BPMN (and, if I'm not wrong, it's basically a easy way to create custom finite state machines) while I'm working on a "lower level".

I have 3 concepts: tasks, links and workflows(a collection of tasks). Links can be created between task of the same workflow or task from different workflows. Workflows can be scheduled based on calendar rules(cron like or "2nd working day of the month" and similar) or external events(api call, new file on s3, ecc ecc). Normally workflows are DAGs, but loops can be created if needed.

The other idea is that it's just the orchestration layer, the task needs to be executed by specific "agent", that can be written in any language (there is a very simple protocol build on top of http to get the tasks from the orchestrator or to send the results) and can be installed on local or remote server.

Just some context: in the mainframe era, businesses used to be orchestrated by a unique "scheduler" running all day long(it was so flexible that basically any business from banking to insurance to manufacturing to industry were able to adapt it to their needs). Bigger companies used to have >100k (or even millions) of tasks per day with very well-defined dependencies. The "good" part was that it was very easy to monitor and manage your batch operations. My idea is to reproduce something similar, but also to add a better support for distributed envs and "human" tasks.