I’ve hand-labeled so many freaking datasets over the years for supervised learning tasks, if only to build out a PoC. Would 100% hand this off to an LLM and just fine tune the results. I definitely do not want an LLM in the loop at inference time for anything I could build a simple classifier for.

Llm labelling is so incredibly useful, as is transforming labelled data to other valid but different forms (like where you might train on sentences but also randomly replace days of the week with others).

You can also do stuff like setup active learning but have the labeller be an llm as well to make things more efficient.

Definitely worth it for PoC level things if not full ones.

Edit - also reviewing, so it can run training, test, review failure cases and investigate itself. You can leave them to plough through a few iterations where the early part is the classic time you’d have a few obviously bad sets of results.