Yes I'll happily extend that to 10x more difficult.

At least compared to building a batched pipeline with SQL. I think you should really think hard whether you really need a streaming pipeline. And even if you find that you do, it may be worthwhile to make a batched pipeline as your first implementation.

I did exactly what you describe in my previous job. In the beginning with reluctance from our architects who wanted to keep banging the dead horse and did not understand the power of SQL "SQL is not real programming, engineers write java" (ok maybe I deserve a straw-man yellow card here, they don't deserve all of that). But I think they understood after a while.

With AWS Athena and Airflow. Good luck, consider me your distant moral support.