It feels like gnu parallel with --transfer-file would have solved this problem