▲ | JackC 3 days ago | |
Cool, I was just looking for something like this! It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was: - load URLs from a file - hand them out to one subprocess per cpu - download them concurrently in threads or async within each subprocess - pull the results back into a single process for formatting and storing Getting this to work and handle queues, ctrl-c, exceptions etc. is just a whole mess involving python builtins created at different times with different interfaces; I hacked until I kind of got it working, but didn't love it. Bundling it all in a single tested package would be great. | ||
▲ | zenapollo 3 days ago | parent [-] | |
I stumbled on grequests for this use case and it just works. |