That's to be expected, in this test setup wall density is the only thing that matters (and Blender physics friction model is probably not quite realistic either). But heavy rolling ball impact isn't the kind load these walls are normally subject to.