Swap is not 'super slow extra memory' (well it is, but that's not useful!), it's a place to temporarily store contents of memory for things less important than what wants to be in fast memory right now. Idle apps can (and are) swapped out to have more file cache even on boxes with large RAM. Memory can also be fragmented and you'll have nominally lots of free space, but in reality allocations will fail; in this case, swap can be used to defragment memory instead of OOM killing stuff.

You always want some swap, even if it's 1GB for a 96GB machine.