Silicone doesn't require plasticizers (because it's elastic on its own) or fire retardants (because it doesn't burn easily). The material itself is also considered biologically inert and is less affected by temperature, solvents, etc. So it's usually the best choice for stuff like that. The reason it's not as common is that it's more expensive and not as durable. It has relatively poor abrasion and cut resistance.

But then, I wouldn't worry about headphones at all. You probably sleep on a mattress made from polyurethane foam that contains plasticizers and fire retardants in much greater quantities. The same goes for your car seats, and they off-gas a lot more when parked in the sun. You'd probably need to eat 1,000 earbuds to match that.