I have very similar experience.

After seeing how people like Andrej Karparthy used vibe coding to generate applications https://x.com/karpathy/status/1903671737780498883?s=61 I realize that

you need to be clear on what you want the LLM to do break down the tasks and give byte sized tasks to llm to do specific thing and sometimes I had to tell it not go and change random files because it found the need to refactor them.