Probably hats were associated with people who work in the sun like farmers or soldiers.. Togas were also impractical clothing compared to a tunic.