> lightweight Swift UI and most logic in a headless cli app could be an interesting design choice

With absolutely no hard data to back this claim, my instinct is that a library would be even better. Anything you can manage to fit into a library can be much more easily tested, and then you can define whatever frontends you want (CLI, SwiftUI, etc.) that wrap it, and even leave the door open for people to implement whatever UI they might want.