So it looks like something that app would be doing, or run another model over the output to smoothen out and remove these things?

Yep, could do simple things like literal regex or all the way up to LLM cleanup, tons of options