The "it's just math" argument may not be technically rigorous, but it's directionally correct. The unstated reasoning invites us to consider why this particular math would be conscious, but not many other forms of math all around us.

First, it seems you've shifted from "intelligent" to "conscious". "These math operations produce consciousness" is different from "these operations produce intelligence".

Second, "it's just math" doesn't mean literally "it's a branch of algebra". It means "it's a computable function". So it can be relevant to the discussion only if you think that intelligence is somehow non-computable, and therefore that there are non-computable processes going on in our brain. Otherwise it's a perfectly pointless remark.