Search code examples
rapidscudf

How to groupby with custom function in python cuDF?


I am new to using GPU for data manipulations, and have been struggling to replicate some of the functions in cuDF. For instance, I want to get a mode value for each group in the dataset. In Pandas it is easily done with custom functions:

df = pd.DataFrame({'group': [1, 2, 2, 1, 3, 1, 2],
                   'value': [10, 10, 30, 20, 20, 10, 30]}

| group | value |
| ----- | ----- |
| 1     | 10    |
| 2     | 10    |
| 2     | 30    |
| 1     | 20    |
| 3     | 20    |
| 1     | 10    |
| 2     | 30    |

def get_mode(customer):
    freq = {}
    for category in customer:
        freq[category] = freq.get(category, 0) + 1
    key = max(freq, key=freq.get)
    return [key, freq[key]]

df.groupby('group').agg(get_mode)

| group | value |
| ----- | ----- |
| 1     | 10    |
| 2     | 30    |
| 3     | 20    |

However, I just can't seem to be able to replicate the same functionality in cuDF. Even though there seems to be a way to do it, of which I have found some examples, but it somehow doesn't work in my case. For example, the following is the function I tried to use for cuDF:

def get_mode(group, mode):
    print(group)
    freq = {}
    for i in range(cuda.threadIdx.x, len(group), cuda.blockDim.x):
        category = group[i]
        freq[category] = freq.get(category, 0) + 1
    mode = max(freq, key=freq.get)
    max_freq = freq[mode]
    
df.groupby('group').apply_grouped(get_mode, incols=['group'],
                                   outcols=dict((mode=np.float64))

Can someone please help me understand what is going wrong here, and how to fix it? Attempting to run the code above throws the following error (hopefully I managed to put it under the spoiler):

Error code
TypingError: Failed in cuda mode pipeline (step: nopython frontend)
Failed in cuda mode pipeline (step: nopython frontend)
- Resolution failure for literal arguments:
No implementation of function Function(<function impl_get at 0x7fa8f0500710>) found for signature:

>>> impl_get(DictType[undefined,undefined]<iv={}>, int64, Literal[int](0))

There are 2 candidate implementations:
    - Of which 1 did not match due to:
    Overload in function 'impl_get': File: numba/typed/dictobject.py: Line 710.
      With argument(s): '(DictType[undefined,undefined]<iv=None>, int64, int64)':
     Rejected as the implementation raised a specific error:
       TypingError: Failed in nopython mode pipeline (step: nopython frontend)
     non-precise type DictType[undefined,undefined]<iv=None>
     During: typing of argument at /opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py (719)
     
     File "../../opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py", line 719:
         def impl(dct, key, default=None):
             castedkey = _cast(key, keyty)
             ^

raised from /opt/conda/lib/python3.7/site-packages/numba/core/typeinfer.py:1086
    - Of which 1 did not match due to:
    Overload in function 'impl_get': File: numba/typed/dictobject.py: Line 710.
      With argument(s): '(DictType[undefined,undefined]<iv={}>, int64, Literal[int](0))':
     Rejected as the implementation raised a specific error:
       TypingError: Failed in nopython mode pipeline (step: nopython frontend)
     non-precise type DictType[undefined,undefined]<iv={}>
     During: typing of argument at /opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py (719)
     
     File "../../opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py", line 719:
         def impl(dct, key, default=None):
             castedkey = _cast(key, keyty)

During: resolving callee type: BoundFunction((<class 'numba.core.types.containers.DictType'>, 'get') for DictType[undefined,undefined]<iv={}>)
During: typing of call at /tmp/ipykernel_33/2595976848.py (6)


File "../../tmp/ipykernel_33/2595976848.py", line 6:
<source missing, REPL/exec in use?>

During: resolving callee type: type(<numba.cuda.compiler.Dispatcher object at 0x7fa8afe49520>)
During: typing of call at <string> (10)


File "<string>", line 10:
<source missing, REPL/exec in use?>

Solution

  • cuDF builds on top of Numba's CUDA target to enable UDFs. This doesn't support using a dictionary in the UDF, but you your use case can expressed with built-in operations with pandas or cuDF by combining value_counts and drop_duplicates.

    import pandas as pd
    ​
    df = pd.DataFrame(
        {
            'group': [1, 2, 2, 1, 3, 1, 2],
            'value': [10, 10, 30, 20, 20, 10, 30]
        }
    )
    ​
    out = (
        df
        .value_counts()
        .reset_index(name="count")
        .sort_values(["group", "count"], ascending=False)
        .drop_duplicates(subset="group", keep="first")
    )
    print(out[["group", "value"]])
       group  value
    4      3     20
    1      2     30
    0      1     10