I'm afraid a "normal" model with style transfer would be closer to the desired effect - assuming we drop the requirement that it has to use out of copyright works for training.

Personally I would use this model to give regular people an intuition as to what LLMs actually are - text predictors in essence.

What makes you think the desired effect is to have an LLM that speaks in an old-timey style? The training process is the whole point.