I would say that 'what not to build' is the second most important decision.

The most important decision is what limitations and constraints you are prepared to accept. There are always limitations and constraints but they are rarely identified up-front.

The most difficult, hard-to-revert decisions are those related to hard constraints imposed by the systems and/or data you have to rely on. Maybe it's not physically possible to obtain the data you need to solve the problem you need to solve for the price you need for it to be viable. Maybe you need to be able to process large files which may take many hours to complete. Or your input data grows father than you can process on a single CPU core. Or the security requirements of a particular piece of data means that you can't efficiently connect it to some external piece of data as you need.