Spark especially is brutal for some reason. Even databrick's AI is bad at spark, which is very funny.

It's probably because spark is so backwards compatible with pandas, but not fully.