Central California has several abandoned asbestos mines which continue to release asbestos into the nearby air and water. I remember considering some hiking around the area once and then came across the warnings regarding exposure.

CA also has a lot of naturally occuring mercury as well. I seem to remember that some lakes in CA are so high in natural(i.e. not from gold mining) mercury that you shouldn't eat the fish.

The Monte Cristo basin in Washington is high enough in Arsenic that you shouldn't drink water there. There are warning signs at trailheads EG the trailhead leading into Gothic Basin and Gothic Peak.

There are several different kinds of asbestos and they differ radically in their level of risk and the worst kinds are not (generally?) found in California. So that signage might just be out of an abundance of caution.

I wouldn't be surprised if by the numbers you have a lot more risk of serious injury driving to and from the hike than from the asbestos, particularly if your hiking doesn't involve intentionally disturbing the ground. :)

For the many posters on HN in the bay area-- watch road cuts for green rocks, asbestos co-occurs with serpentinite all over the coast range-- and serpentinite is the state rock! You can easily find some with asbestos, usually grey/white fibrous strands on the rocks. Don't grind it up and huff it, but you can thrill your friends by showing them some boogieman-mineral you found.

(It's only fitting that the state rock is technically known to the state of california to cause cancer... :P)