I think this purely as an idea is pretty fun, and there is value in that. But beyond the initial impressions it's exactly as you say. It's not different at all from others in how it will get annoying over time.

Accessibility is a big concern with all kinds of CAPTCHAs it seems. Even without any disabilities, I've seen some that I cannot solve because it's illegible.