Search code examples
pythonalgorithmmachine-learningprecision-recall

Evaluation of lists: AvgP@K and R@K are they same?


My goal is to understand Average Precision at K, and Recall at K. I have two lists, one is predicted and other is actual (ground truth)

lets call these two lists as predicted and actual. Now I want to do precision@k and recall@k.

Using python I implemented Avg precision at K as follows:

def apk(actual, predicted, k=10):
    """
    Computes the average precision at k.

    This function computes the average precision at k between two lists of items.

    Parameters
    ----------
    actual: list
            A list of elements that are to be predicted (order doesn't matter)
    predicted : list
            A list of predicted elements (order does matter)
    k: int, optional

    Returns
    -------
    score : double
            The average precision at k over the input lists

    """
    if len(predicted) > k:
        predicted = predicted[:k]

    score = 0.0
    num_hits = 0.0

    for i,p in enumerate(predicted):
        if p in actual and p not in predicted[:i]:
            num_hits += 1.0
            score += num_hits / (i + 1.0)

    if not actual:
        return 1.0
    if min(len(actual), k) == 0:
        return 0.0
    else:
        return score / min(len(actual), k)

lets assume that our predicted has 5 strings in following order: predicted = ['b','c','a','e','d'] andactual = ['a','b','e']since we are doing @k would the precision@k is same asrecall@k? If not how would I dorecall@k`

If I want to do f-measure (f-score) what would be the best route to do for above mention list?


Solution

  • I guess, you've already checked wiki. Based on its formula, the 3rd and the biggest one (after the words 'This finite sum is equivalent to:'), let's see at your example for each iteration:

    1. i=1 p = 1
    2. i=2 rel = 0
    3. i=3 p = 2/3
    4. i=4 p = 3/4
    5. i=5 rel = 0

    So, avp@4 = avp@5 = (1 + 0.66 + 0.75) / 3 = 0.805; avp@3 = (1 + 0.66) / 3 and so on.

    Recall@5 = Recall@4 = 3/3 = 1; Recall@3 = 2/3; Recall@2 =Recall@1 = 1/3

    Below is the code for precision@k and recall@k. I kept your notation, while it seems to be more common to use actual for observed/returned value and expected for ground truth (see for example JUnit defaults).

    def precision(actual, predicted, k):
        act_set = set(actual)
        pred_set = set(predicted[:k])
        result = len(act_set & pred_set) / float(k)
        return result
    
    def recall(actual, predicted, k):
        act_set = set(actual)
        pred_set = set(predicted[:k])
        result = len(act_set & pred_set) / float(len(act_set))
        return result