They never say that their hold-out dataset to evaluate perplexity is just in English. Or written text for that matter (could also include code, ...)