May I humbly submit:

Automatologist: One who studies the behavior, adaptation, and failure modes of artificial agents and automated systems.

Automatology: the scientific study of artificial agents and automated-system behavior.

Greek word derivatives all seem to be a bit unwieldy; Latin might work better.

While the names aren't set yet, the field of study is apparently already being pushed forward. [1]

[1] https://www.quantamagazine.org/the-anthropologist-of-artific...