use serverless then. its literally the simplest solution. what sort of poor decisions your team is making? it needs to run 10 seconds but still spinning up a cluster?

Databricks serverless has tons of limitations, a simple server would suit our needs much better