Maybe the best compromise here is a markdown interpreter that converts to latex so you can use existing libraries and templates but not have to worry as much about syntax.

Pandoc can do this [0], and there's also the "markdown" LaTeX package [1].

[0]: https://pandoc.org/MANUAL.html#specifying-formats

[1]: https://ctan.org/pkg/markdown

I don't think syntax is the problem here. Using some tool that transpiles to LaTeX, you'll still have to fight your TeX engine sometimes, but now you also add friction and gotchas of your tool that does transpiling.

I've tried writing Org and compiling it to LaTeX, but gave up this idea because ultimately I had to figure out idiosyncrasies of both LaTeX and Org-mode engines.

There are already good html to pdf solutions, people using latex generally would not be satisfied with those