Are you running vacuums at all? auto_vacuum enabled at all?

https://sqlite.org/lang_vacuum.html

In memory DBs don't have anything to vacuum.

However... what you (and OP) are looking for might be pragma shrink_memory [1].

[1] https://sqlite.org/pragma.html#pragma_shrink_memory