How hard would it be to fix this? Could we theoretically add or subtract enough material or make whole thing slightly more dense or less dense to compensate?

Per Wikipedia, the discrepancy is approximately 74 km, so digging a ditch with an average depth of approximately 74/2π ≅ 12 km around the circumference of the Earth would theoretically fix the problem.

Feasibility and geological implications are left as exercises for the reader.

Regardless, I suspect a more cost-effective fix would be to redefine the meter to be a couple "legacy" millimeters longer.