My company is heavily invested in Databricks and let me tell you it sucks. 5 min to spin up a job that needs to run for 10 seconds is a terrible way to spend ones time and money.

Yes, but you can pay to spin up unlimited numbers of jobs that spend 97% of their time spinning up! Truly cloud scale.

use serverless then. its literally the simplest solution. what sort of poor decisions your team is making? it needs to run 10 seconds but still spinning up a cluster?