Could not able to insert properly if items crosses 1350+ in local dynamodb

Question

I am using local docker setup for dynamodb. I could able to insert items when items are less it starts behaving wrong when data items goes beyond 1350 around.

table = dynamodb.create_table(
        TableName='logevents',
        KeySchema=[
            {
                'AttributeName': 'Id',
                'KeyType': 'HASH'  # Partition key
            }
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'Id',
                'AttributeType': 'S'
            }
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 1,
            'WriteCapacityUnits': 1
        }
    )

item = {
    "log_datetime": "2019-08-04 16:16:24:302000",
    "Action": "ApiCallFailed",
    "intuit_tid": "f9d63462-51a6-463b-a19f-b5c6c7867388",
    "PackageName": "com.intuit.billingcomm.billing.ius.facade.DefaultIUSServiceFacade.fetchAllGrants",
    "FileName": "DefaultIUSServiceFacade.java",
    "PID": "9",
    "Processed_timestamp": "2019-09-05 18:17:06:515970",
    "APIKey": "GetAllGrants",
    "thread_id": "x1B35mhttps-jsse-nio-8443-exec-71\\x1B0;39m",
    "LastMessage": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX is currently unavailable: status=500",
    "Exceptions": [
        "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.MessageProcessingException",
        "com.intuit.billingcomm.billing.exception.IAMTicketClientException",
        "com.intuit.platform.integration.hats.common.XXXXXXXXXXXXXXXXX"
    ],
    "Event": "XXXXXXXXXX",
    "CategoryLevel1": "XXXXXXXXXXX",

    "Domain": "IUS",
    "LineNo": "128",
    "CategoryLevel2": "TicketServerFailure"
    }

    for x in range(6000):
        item['Id']=str(x)
        table.put_item(
            Item=item)
    scan = table.scan(
        ProjectionExpression='#k',
        ExpressionAttributeNames={
            '#k': 'Id'
        }
    )
    print(str(len(scan['Items'])))

If i give range 600 it shows data correctly but if i give 6000 it shows no of items 1332. is there any restriction?

Answer 1

I believe that the likely scenario is that you got a paginated resulted: when the size of the scanned items cross a certain threshold you get only the the first few (in your case, the first 1332). you then need to issue subsequent calls to scan() to get the next chunks.

In each call (other than the first one) you need to set ExeclusiveStartKey with the value of LastEvaluatedKey from the response. If LastEvaluatedKey is not set, you should stop calling scan(). See sample code from this answer.