I can only imagine what that does to the poor SSD

Pretty much nothing, if you only read the weights from SSDs, i.e. if you keep any writable caches in your actual DRAM.

Paging in experts is mostly reading so on the first order effects it would be fine. These might be second order effects from e.g. write caches needing to be flushed more often (and maybe swapping other applications, if you use a swap file or partition) but it probably wouldn't be too much of an issue.