There's no particular reason to use RGB for this kind of machine vision - cognition problem either.

Infra-red of a few different wavelengths as well as optical light ranges seems like it'd give a superior result?

You've just described some of the rationale for using LIDAR.