I mean we've been using technology to solve long-standing problems in mathematics that no human had been able to do for years. Is Deep Blue sentient because it can beat any human at chess?

Deep Blue's strength was leveraging massive compute to execute a task-specific, human-written algorithm. The problems which LLMs are tackling don't elude mathematicians because they require too much number-crunching, but because they demand creative problem solving. The latter seems more profound, even if it doesn't imply sentience.