Lambda doesn't have permission to add tags to managed instances in systems manager

Question

I have a multi account structure in AWS, where I have a master and child accounts. I am following this guide in order to propagate tags from the child instances to the master account, once they have been activated and I can manage the instances in the master account (systems manager).

So far it all works to the point where the lambda in the master account has all of the tags it needs. However, it is unable to add the tags to the managed instances in systems manager. Not sure why the role still can't access the tags, given the permissions...

This is the error I get:

[ERROR] 2019-03-29T09:14:02.419Z a00a68ba-9904-4199-bcae-cad75f6f5232 An error occurred (ValidationException) when calling the AddTagsToResource operation: Caller is an end user and not allowed to mutate system tags instanceId: mi-0d3bfce27d073c0f2

This is the lambda function with the attached role:

AWSTemplateFormatVersion: '2010-09-09'
Description: Management function that copies tags
Resources:
  rSSMTagManagerRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: Automation-SSMTagManagerRole
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "lambda.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/aws/"
      Policies:
        - PolicyName: "CopyInstanceTagsToSSMPolicy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                - ssm:AddTagsToResource
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
                - tag:*
                Resource: "*"

  fnSSMTagManager:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: Automation-SSM-Tag-Manager
      Handler: index.lambda_handler
      Role: !GetAtt [rSSMTagManagerRole, Arn]
      Description: >
        Copies tags from the list of instances in the event
        context to the specified managed instances.
      Code:
        ZipFile: |+

          import boto3
          import json
          import logging

          #setup simple logging for INFO
          logger = logging.getLogger()
          logger.setLevel( logging.WARN )

          client = boto3.client( 'ssm' )

          def lambda_handler( event, context ):
              """Copies tags from the list of instances in the event
              context to the specified managed instances.
              """
              for instance in event[ "instances" ]:
                 addTags( instance[ "instanceId" ], instance[ "tags" ] )

          def addTags( resourceid, tags ):
              logger.info( "Configuring " + resourceid + " with " + str(tags) )
              try:
                  response = client.add_tags_to_resource(
                      ResourceType='ManagedInstance',
                      ResourceId=resourceid,
                      Tags=tags
                  )
                  logger.info( response )
                  return response
              except Exception as e:
                  errorMessage = str(e) + "instanceId: " + resourceid
                  logger.error( errorMessage )
                  return errorMessage

      Runtime: python3.6
      Timeout: '90'

Answer 1

Using the same guide. Faced the exact same error. It turned out that the instances in the agency account were having too many(10 plus) tags which caused the Tag Manager to give this error. Modified the Tag collector lambda function to propagate only specific tags instead of all tags. That cleared the error.