Can't one just remove the signature and re-sign it with anything else after tampering? Who verifies PDFs that hard?

If you're performing OCR, you're almost by definition, disregarding the source file. The whole point of OCR is to be transformative.