I thought I read that in general it’s just better to decline contrast because it doesn’t actually add value to the scan.

It depends on the indication for the scan. Some indications do not require contrast, others MUST have contrast in order to have any value. If you refuse contrast without understanding the reason, you may be simply wasting your time and money.