Feels off that only 5 months ago the company just migrated to DuckDB only to now move to something else.

Who did the growth estimation of the data? This looks like an example of bad planning.