If I wasn't paid, I wouldn't work for them.

While this is true for most of the cases, it's not the case always. There are definitely jobs I'd work for less than I make at the job now and there are jobs which are considered "dream jobs", but ultimately there needs to be a solid base to make you and your family feel comfortable if you're spending at least 1/3 of your day on.