When to use S3 API pagination

Question

I'm using the boto3 client to access date stored in an Amazon S3 bucket. After reading the docs, I see that I can make a request with this code:

s3 = boto3.resource('s3')
bucket = s3.Bucket(TARGET_BUCKET)

for obj in bucket.objects.filter(
  Bucket = TARGET_BUCKET,
  Prefix = TARGET_KEYS + KEY_SEPARATOR
):
  print(obj)

I test against a bucket where I've stored 3000 objects and this fragment of code retrieves the references to all the objects. I've read that all the API calls to S3 return at most 1000 entries.

But reading the boto3 documentation, paginator section, I see that some S3 operations need to use pagination to retrieve all the results. I don't understand why the upper code works unless the code is using the paginator under the hood. And this is my question, can I safely assume that the upper code always will retrieve all the results.?

Answer 1

According to the documentation here, the pagination is handled for you.

A collection provides an iterable interface to a group of resources. Collections behave similarly to Django QuerySets and expose a similar API. A collection seamlessly handles pagination for you, making it possible to easily iterate over all items from all pages of data.