Python Joblib Module
Joblib is a set of tools to provide lightweight pipelining in Python. It is particularly optimized for fast cloning of objects (like large NumPy arrays) and easy parallel computing.
Installation
pip install joblibCaching (Memoization)
Joblib provides a simple way to cache the results of functions to disk. This is useful for expensive computations.
from joblib import Memory
location = './cachedir'
memory = Memory(location, verbose=0)
@memory.cache
def expensive_function(x):
print(f"Computing {x}...")
return x * x
print(expensive_function(2)) # Computes
print(expensive_function(2)) # Returns cached resultParallelism
Joblib provides a simple helper class Parallel to write parallel for loops using multiprocessing.
from joblib import Parallel, delayed
import math
# A function to be executed in parallel
def sqrt_func(i):
return math.sqrt(i)
# Run 2 jobs in parallel
results = Parallel(n_jobs=2)(delayed(sqrt_func)(i) for i in range(10))
print(results)Serialization
Joblib's dump and load functions are a replacement for pickle that work efficiently on arbitrary Python objects containing large data, particularly NumPy arrays.
import joblib
import numpy as np
data = np.random.random((1000, 1000))
# Save to file
joblib.dump(data, 'data.joblib')
# Load from file
loaded_data = joblib.load('data.joblib')