MTurk: Best 2 out of 3 HIT creation/retrieval

Question

Desired Result

I can retrieve the agreed upon answer value for a given HIT

The Problem

When I call

CLIENT.list_review_policy_results_for_hit(HITId=hit['HITId'])

to retrieve the results for a specific HITId from the mturk sandbox endpoint, the "Value" for the agreed answer is an empty string. I assume I am missing some sort of question mapping in the layout definition or create_hit() logic but the docs are really unclear on how this should work so I'm not even sure "list_review_policy_results_for_hit()" is the correct method for retrieving results.

Background

I'm working on creating a pipeline for uploading single question HITs where the turker is expected to rate the text on a binary scale as "quality" or "not quality". I want to evaluate each HIT with a "best 2 out of 3" ruling where the HIT is served to two turkers for review and an additional third turker if a tie-breaker is necessary.

I'm using the SimplePlurality/2011-09-01 review policy to handle the "2 out of 3" logic.

I am happy to include more information/code samples upon request but didn't want to bloat the initial post with unnecessary information.

Answer 1

It's possible that you haven't configured your ReviewPolicy correctly.

Can you please share the ReviewPolicy you're passing in as the HITReviewPolicy parameter in your CreateHIT request?

I've shared an example below. You would replace "your_question_id" with the identifier from your HIT.

Thank you

{
    'PolicyName': 'SimplePlurality/2011-09-01',
    'Parameters': [
        {
            'Key': 'QuestionIds',
            'Values': ['your_question_id']
        },
        {
            'Key': 'QuestionAgreementThreshold',
            'Values': ['66']
        },
        {
            'Key': 'ExtendIfHITAgreementScoreIsLessThan',
            'Values': ['100']
        },
        {
            'Key': 'ExtendMaximumAssignments',
            'Values': ['3']
        },
        {
            'Key': 'ExtendMinimumTimeInSeconds',
            'Values': ['86400']
        },
        {
            'Key': 'DisregardAssignmentIfRejected',
            'Values': ['T']
        }
    ]
}