If memory is paged out we could defer cleaning out garbage from it.

I'm not aware of any way to find that out as an application. Databases can do this via the page cache since that's basically an implementation of swapping.