“Claude, spend the next 10 hours trying to solve the Reimann Hypothesis”.

I agree that incentives are misaligned but there’s several competing model providers. If one gets funny with their costs people will jump ship, especially if the gap between the top 2 labs and everyone else keeps shrinking.

“I can take current sources and tell you how solved this is, but I am not willing to work to a timeframe or to solve things that aren’t yet solved by mathematicians or science”

These safeguards already exist when they get a whiff that you might be using Claude to fix security issues. Doesn’t seem farfetched given the incentives I outlined that they would apply to this kind of abuse.

How loose those controls are becomes a market force.