Search code examples
pythonpandascountfrequencyfrequency-analysis

Python Pandas count most frequent occurrences


This is my sample data frame with data about orders:

import pandas as pd
my_dict = { 
     'status' : ["a", "b", "c", "d", "a","a", "d"],
     'city' : ["London","Berlin","Paris", "Berlin", "Boston", "Paris", "Boston"],
     'components': ["a01, a02, b01, b07, b08, с03, d07, e05, e06", 
                    "a01, b02, b35, b68, с43, d02, d07, e04, e05, e08", 
                    "a02, a05, b08, с03, d02, d06, e04, e05, e06", 
                    "a03, a26, a28, a53, b08, с03, d02, f01, f24", 
                    "a01, a28, a46, b37, с43, d06, e04, e05, f02", 
                    "a02, a05, b35, b68, с43, d02, d07, e04, e05, e08", 
                    "a02, a03, b08, b68, с43, d06, d07, e04, e05, e08"]
}
df = pd.DataFrame(my_dict)
df

enter image description here

I need to count most frequent:

  1. Top-n co-occurring components in orders
  2. Top-n most frequent components (regardless of co-occurrence)

What will be the best way to do it?

I can see the relation to market basket analysis problem as well, but not sure how to do it.


Solution

  • @ScottBoston's answer shows vectorized (hence probably faster) ways to achieve this.

    Top occurring

    from collections import Counter
    from itertools import chain
    
    n = 3
    individual_components = chain.from_iterable(df['components'].str.split(', '))
    counter = Counter(individual_components)
    print(counter.most_common(n))
    # [('e05', 6), ('e04', 5), ('a02', 4)]
    


    Top-n co-occuring

    Note that I'm using n twice, once for "the size of the co-occurrence" and once for the "top-n" part. Obviously, you can use 2 different variables.

    from collections import Counter
    from itertools import combinations
    
    n = 3
    individual_components = []
    for components in df['components']:
        order_components = sorted(components.split(', '))
        individual_components.extend(combinations(order_components, n))
    counter = Counter(individual_components)
    print(counter.most_common(n))
    # [(('e04', 'e05', 'с43'), 4), (('a02', 'b08', 'e05'), 3), (('a02', 'd07', 'e05'), 3)]