Definitely take the job, amazon is not an endgame place to work but its a really good stepping stone

You learn a ton of stuff there as a new grad

CS masters degrees are not really a differentiator in the job market, in my experience the exact same jobs are filled by people with bachelors/masters

Employers are not like "we really need someone with a masters, get me someone with a masters"

I would only do grad school if you actually want to do research/applied research

Keep in mind that a lot of the people on this website are late in their careers (and bored by them) and take having a job for granted so doing a masters sounds more exciting to them.