They'll never give a clear definition of intelligence because if they did their claims could be falsified. Qualifying what "intelligence" can do in a formal sense is actually a very well-studied field called computational complexity theory. Computational complexity theory shows than many many real world problems and processes cannot be solved/simulated much better without an exponential increase in computational power, regardless of the program/"intelligence" used. Singulatarian cultists want you to believe that lower bound complexity classes don't exist, which is mathematically equivalent to telling you that AI can somehow magically make 1+1=3.

It would also require quite sophisticated and careful thinking about the stuff e.g. Merleu-Ponty and Derrida did, and paying close attention to the last thirty years or so of neuroscience and biology.

One problem they'd have to grapple with is that human intelligence is embodied and carries the same complexity as physical matter does, and software does not since it is projected onto bit processing logic gates. If they really want to simulate embodied intelligence, then it is likely to be excruciatingly slow and resource intensive.

It would be cheaper and more efficient to get humans to become more like computers.