I'm attempting to accelerate a piece of Python code using Numba. Within this code, I'm also employing mpi4py for parallelization. However, I've encountered an error. I've attempted to provide a minimal error-reproducing example below:
import numpy as np
from numba import njit, prange
from mpi4py import MPI
#####MPI setting#####
comm=MPI.COMM_WORLD
rank=comm.Get_rank()
size=comm.Get_size()
N_theta_to_scan=1000
n_per_proc=int(N_theta_to_scan/size)
n_more=int(N_theta_to_scan%size)
if rank<n_more:
start=rank*(n_per_proc+1)
number_to_cal=n_per_proc+1
else:
start=n_more*(n_per_proc+1)+(rank-n_more)*n_per_proc
number_to_cal=n_per_proc
#####function#####
@njit(parallel=True)
def func1(na,nb,nc):
to_sum=np.zeros(na*nb*nc)
for a in prange(0,na):
for b in prange(0,nb):
for c in prange(0,nc):
to_sum[a*nb*nc+b*nc+c]=a*b*c
out=np.sum(to_sum)
return out
@njit(parallel=True)
def func2(start,number_to_cal):
to_sum=np.zeros(number_to_cal)
for i in prange(start,start+number_to_cal):
to_sum[i-start]=func1(i,i*i,i*i*i)
out2=np.sum(to_sum)
return out2
#####main section#####
to_be_gather=np.array([func2(start,number_to_cal)])
gatheres=np.zeros(0)
comm.Reduce(to_be_gather,gatheres,op=MPI.SUM)
The error occurs for this segment of the code:
MemoryError: Allocation failed (probably too large).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/share/workspace/wuliang/hanlin/test/test.py", line 38, in <module>
to_be_gather=func2(start,number_to_cal)
^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: CPUDispatcher(<function func2 at 0x2aed19d7b420>) returned a result with an exception set
Next, I attempted to remove func1() and modified func2() by replacing to_sum[i-start]=func1(i,i*i,i*i*i) with to_sum[i-start]=i, which resulted in successful execution. Additionally, I tried running the code with only rank 0 executing the main function, i.e., running the code under the condition if rank==0:, which also ran successfully. I'm wondering where the error lies, and what modifications should I make to achieve the functionality of the current code?
In reference to the code of original question and to comments regarding coupling Numba and MPI; using MPI from within Numba compiled code (also with parallel=True) is possible with the numba-mpi package.
Here's a compute-pi-in-parallel example from the package README where a reduction is performed within @numba.jit decorated function (using mpi4py, the reduction needs to be done outside of the JIT-compiled code, as in the question):
import numba, numpy as np, numba_mpi
N_TIMES = 10000
@numba.jit
def get_pi_part(n_intervals=1000000, rank=0, size=1):
h = 1 / n_intervals
partial_sum = 0.0
for i in range(rank + 1, n_intervals, size):
x = h * (i - 0.5)
partial_sum += 4 / (1 + x**2)
return h * partial_sum
@numba.jit
def pi_numba_mpi(n_intervals):
pi = np.array([0.])
part = np.empty_like(pi)
for _ in range(N_TIMES):
part[0] = get_pi_part(n_intervals, numba_mpi.rank(), numba_mpi.size())
numba_mpi.allreduce(part, pi, numba_mpi.Operator.SUM)
The package README includes a performance comparison with mpi4py.