Python Multiprocessing Module
The multiprocessing module allows you to create processes that run independently. Each process has its own Python interpreter and memory space, bypassing the Global Interpreter Lock (GIL). This makes it ideal for CPU-bound tasks.
Importing the Module
import multiprocessingThe Process Class
To create a process, you create an instance of the Process class and pass it the function you want to run.
import multiprocessing
import time
def worker(num):
print(f'Worker: {num}')
time.sleep(1)
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target=worker, args=(i,))
jobs.append(p)
p.start()
for p in jobs:
p.join()The Pool Class
The Pool class offers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism).
import multiprocessing
def square(x):
return x * x
if __name__ == '__main__':
with multiprocessing.Pool(processes=4) as pool:
results = pool.map(square, [1, 2, 3, 4, 5])
print(results)Inter-Process Communication
Queue
The Queue class is a thread and process safe queue.
import multiprocessing
def producer(q):
q.put('hello')
def consumer(q):
print(q.get())
if __name__ == '__main__':
q = multiprocessing.Queue()
p1 = multiprocessing.Process(target=producer, args=(q,))
p2 = multiprocessing.Process(target=consumer, args=(q,))
p1.start()
p2.start()
p1.join()
p2.join()Sharing State
Shared Memory
Data can be stored in a shared memory map using Value or Array.
import multiprocessing
def f(n, a):
n.value = 3.14159
for i in range(len(a)):
a[i] = -a[i]
if __name__ == '__main__':
num = multiprocessing.Value('d', 0.0)
arr = multiprocessing.Array('i', range(10))
p = multiprocessing.Process(target=f, args=(num, arr))
p.start()
p.join()
print(num.value)
print(arr[:])