TIL: Python Threadpool

04 Jul 2016

til
python

I’m not very proficient in Python yet, but while playing with it, I felt the need to do a bunch of API calls concurrently.

In Ruby, you can achieve that with gems like concurrent-ruby, but turns out Python has an implementation of a threadpool in its standard library, and you can use it like this:

from multiprocessing.pool import ThreadPool

# Work that will be done concurrently
def work(index):
    # some long process
    return index

pool = ThreadPool(processes=5)
results = []

for index in range(0, 10):
    # Post `work` in the pool
    result = pool.apply_async(work, (index,))
    results.append(result)

for result in results:
    # wait for each result
    result.get()