Did you validate this by running a A/B test? Main question is were you able to classify back into your known categories correctly all the time, or did the errors compound from the llm hallucination plus embedding search

Using a Nano model, a tad worse than shipping a vocabulary to a larger OpenAI model. (And it’s an huge improvement on not classifying the queries at all).

But no classification is perfect. In search in particular, you will also want to have places for manual intervention for high priority queries.