This is a longstanding UX issue with unlisted URL’s. I’ve encountered this in my own products. Folks don’t know that sharing a link means it definitely will get indexed by Google.

Unfortunately this can’t be fixed via robots.txt — Google won’t proactively crawl the URL, but it does still index pages it finds from external links.

A `noindex` meta tag, however, does prevent pages from being indexed, so that feels like the appropriate solution here.

you can also do it via http response header, feels cleaner to me