Currently building something like this for my city. One major challenge I've come across so far is that most APIs will give you coordinates for an address of a place within a building but none of the free, paid or crowdsourced options have reliable information on outside seating polygons. Of course, you could always display the places around you with live shadow data on a map, leaving it up to the user to zoom in and decide based on the satellite image whether the restaurant or café offers outside seating. But to plot the route and then suggest nearby sun/shadow seating options to the user, you would need this information.

You'd need stereo fotos from a low flying plane for this. Several cities are doing this for decades already. I've built such a 3D map for my city like 25 years ago.