If you randomly position pieces on the board and then ask the LLM to play chess, where each piece still moves according to its normal rules, does it know how to play still?

If the goal is to create a model that simulates normal human intelligence, yes. Some try to measure accuracy or performance based on expertise though.