▲ | juicypt 4 days ago | |
Working on a small SQL-based streaming ETL tool. It reads from a replication stream and allows you to trim/enrich the replicated data by running SQL queries from the database, then writing the result out to another database (also using a custom SQL query, so it's easy to do upserts or joining with other data on the destination database). It's working really well, and I'm just sprucing up logging and documentation a bit before making the source code public on github. The idea is for it to be a much simpler alternative to things like Debezium for small to medium sized projects. Currently supports postgres for input, and postgres and clickhouse for output with more databases coming down the road. |