Unable to list all secrets from AWS secret manager

Question

I am trying to list all secrets available in AWS secret manager using lambda function, following is the python code snippet;

region='us-west-2'

session= boto3.sesssion.Session(region_name=region)

client = session.client('secretsmanager')

secrets =  client.list_secrets()

secrets_manager = secrets['SecretList']

for secret in secrets_manager:
  print(secret['Name'])

Above code only lists few secret not all the secrets but running following CLI command returns all secrets.

aws secretsmanager list-secrets | grep "Name"

What am I missing in python code? Please advise

Answer 1

The API is paginated. You need to send multiple requests to get all pages of responses. The CLI does this for you by default.

The easiest way is to use the paginator API in boto3 -- it will correctly implement pagination for you (which may be slightly different between different AWS services/APIs)

client = session.client('secretsmanager')
paginator = client.get_paginator('list_secrets')
page_iterator = paginator.paginate()
for page in page_iterator:
    print(page)

Or you can do this 'manually' for the same effect:

secrets = []

response = client.list_secrets()
secrets.extend(response['SecretList'])
while 'NextToken' in response:
    response = client.list_secrets(NextToken=response['NextToken'])
    secrets.extend(response['SecretList'])
for secret in secrets:
    print(secret['Name'])