Check out using maybe some kind of monocular depth estimation models, like Apple's Depth Pro (https://github.com/apple/ml-depth-pro) and use the depth map to predict a path?

Very cool project though!

Thank you! We'll check it out! Yeah building a map might be the way to go. With E2E its quite hard to ensure "intelligent" cleaning