I mean that it leaves us with the co2 absorbing part already ground up with metals removed.

Aren't the metals the CO₂-absorbing part? Olivine is the fayalite–forsterite–tephroite continuum. Fayalite is Fe₂SiO₄. Forsterite is Mg₂SiO₄. Tephroite is Mn₂SiO₄. (Admittedly, as metals go, those aren't very toxic.)

If you remove the metals from olivine what's left over is silica. Like, quartz. It doesn't absorb CO₂.