I am not concerned about being a meat proxy (fine).I worry about the possible "unfounded oracle." A model that although constrained by humans, (like delphi) acts without any human criticism. Like multivac from Isaac Asimov's short story:

https://www.commonlit.org/texts/the-machine-that-won-the-war