What's the reason of incompatibility of Islam and Jewish religion?

Nothing in most of their beliefs. They’re both monotheistic and similar in many regards as Islam largely inherited its tenants from both Judaism and Christianity.

Jews were often well treated and flourished in the earlier Islamic caliphates.

But with the formation of a Jewish Israel the conflict. Generally in Islamic belief there must be an Islamic caliphate with Sharia Law. Jerusalem is considered one of the holy sites of Islam and therefore belongs to that caliphate.

That’s contrasted with Judaism and Israel being the land promised to the Jews. Though modern Israel was largely founded by secular Jews so it’s a bit more complicated on that front.