I love good programming languages, if that counts for anything, and I think there's a huge amount of room and value in "using an LLM to translate one programming language to another".

I also think your instincts are on a useful path. Perhaps using a very small and stupid LLM to generate plausible-sounding-but-probably-wrong programs might be a path to generating interesting test cases?