Add filter expressions to paginated Boto3 dynamodb scan

Question

I am attempting to filter a paginated scan request to a dynamodb table.

While a traditional scan filter would use something like the following:

response = table.scan(
        FilterExpression=Attr("Date").eq(date)
    )

Attempting to pass this to the paginator scan however:

dynamodb = boto3.client('dynamodb')
dynamores= boto3.resource('dynamodb')
table = dynamores.Table('table')

pagination_config = {"MaxItems": 2, "PageSize": 2}

paginator = dynamodb.get_paginator('scan')
response_iterator = paginator.paginate(
                TableName=table.table_name, 
                PaginationConfig=pagination_config,
                FilterExpression=Attr("Email").contains(user) &
                    Attr("Category").contains(types) 
            )

Results in a parameter validation error

botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter FilterExpression, value: <boto3.dynamodb.conditions.And object at 0x7f550f4b1940>, type: <class 'boto3.dynamodb.conditions.And'>, valid types: <class 'str'>

I have tried filtering on different keys without success and the response remains the same, even when filtering on boolean or integer columns. Is there a way to filter the scan using the operation_parameters argument?

I have also attempted using JMES filtering without success as the followiing:

response_iterator = paginator.paginate(
                TableName=table.table_name, 
                PaginationConfig=pagination_config
            )

filtered_iterator = response_iterator.search("Contents[?contains(Category, 'File Status')]")

Yields no results despite there being an entry matching this criteria in the table.

Answer 1

You are treating the paginator like the Resource client which it is not. You need to be more low level like this example:

https://github.com/aws-samples/aws-dynamodb-examples/blob/master/DynamoDB-SDK-Examples/python/WorkingWithScans/scan_with_paginator.py