The upside of starting with Consider Phlebas is that the story doesn't really take place within the Culture. It provides you with an outside perspective. It also sets up some things that are referenced in later books.

The downside is that it's quite a bit different from (and imo a bit worse than) the other books and could mislead you about what the series is like - or even turn you off entirely.

If you're already committed to reading the entire series, I'd start with Consider Phlebas. If you're unsure, start with The Player of Games.

I actually really like Consider Phlebas - probably because I read it in one go in a day off I allowed myself between finishing my final year project and starting revision for my finals. This was in 1988 <sigh>