Massively simplifying here but when doing one rasterization pass you process a flat list of triangles, in order, once. When raytracing a camera view you query a database of triangles, at least once for each pixel. To avoid scanning the whole list of triangles for every pixel, the database needs a fancy 3D spatial index whose structure depends on the exact position of each vertex. When you modify the position of vertices you have to fix up or rebuild the fancy spatial index too.