I wish there was more research (maybe philosophy) would go into characterizing consciousness and intelligence, so that we could at least define what we are missing in current AI systems.

Philosophy of consciousness is at least 2,500 years old.

There’s been plenty of philosophy, but what will probably happen is a re-definition of the terms to a more rigorous and repeatable mathematical formulation. This will fail to satisfy philosophical or fundamental questions, but will enable better quantitative predictions.

This essentially is where the schism is between science and philosophy, and has played out repeatedly across history. Heat for example was redefined to a specific physical property, and subjective experiences of warmth were then explored in reference to that. Or look back to the moment when Newton essentially said “I don’t know what gravity is, but I can accurately calculate any ballistic trajectory you can think of.”

But if they'll never be conscious, wouldn't that not be a terrific waste of talent and effort?