Etymologically you're correct that semites refers to [all] people from middle east and horn of Africa. But the label is from the European perspective, where it was used to refer to Jews. I'm sure that if they'd had a say in it, they would not have referred to themselves as Jews.

But yeah, there is indeed some irony in the term "antisemitism" in the context you describe