Python Concurrent Futures Module
The concurrent.futures module provides a high-level interface for asynchronously executing callables. It abstracts the execution of tasks in threads or processes using ThreadPoolExecutor and ProcessPoolExecutor.
Importing the Module
import concurrent.futuresExecutor Objects
ThreadPoolExecutor
ThreadPoolExecutor is an Executor subclass that uses a pool of threads to execute calls asynchronously. It is suitable for I/O-bound tasks.
import concurrent.futures
import time
def task(n):
print(f"Processing {n}")
time.sleep(1)
return n * n
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
future = executor.submit(task, 5)
print(f"Result: {future.result()}")ProcessPoolExecutor
ProcessPoolExecutor is an Executor subclass that uses a pool of processes to execute calls asynchronously. It uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock (GIL) but also means that only picklable objects can be executed and returned. It is suitable for CPU-bound tasks.
import concurrent.futures
def square(n):
return n * n
if __name__ == '__main__':
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(square, [1, 2, 3, 4, 5])
for result in results:
print(result)Submitting Tasks
submit()
The submit() method schedules the callable, fn, to be executed as fn(*args, **kwargs) and returns a Future object representing the execution of the callable.
with concurrent.futures.ThreadPoolExecutor() as executor:
future1 = executor.submit(pow, 323, 1235)
future2 = executor.submit(pow, 323, 1235)
print(future1.result())
print(future2.result())map()
The map() method is similar to the built-in map() function. It runs the function asynchronously on the iterables.
with concurrent.futures.ThreadPoolExecutor() as executor:
for result in executor.map(lambda x: x**2, range(10)):
print(result)Future Objects
A Future instance represents the encapsulation of the asynchronous execution of a callable.
result(timeout=None): Return the value returned by the call.cancel(): Attempt to cancel the call.done(): ReturnTrueif the call was successfully cancelled or finished running.add_done_callback(fn): Attaches the callablefnto the future.
as_completed()
The as_completed() function returns an iterator over the Future instances (that are possibly created by different Executor instances) that yields futures as they complete (finished or cancelled).