Search code examples
pythonyamltroposphere

Stacker AwsvpcConfiguration passed in as yaml and recieved as str

I'm trying to get a Stacker project (uses Troposphere under the covers) up and running and I've been able to create several resources but am failing to figure the subnet piece out. I've tried to pass them into my blueprint class several different ways but I think this one is closest and it's similar to the other implementations that don't outright explode. The idea is to accept some configuration for a subnet like this.

In my my_cluster.yaml, I have:

 stacks:
 ...
  - name: cluster-networks
    description: "Networks for an ECS cluster"
    class_path: blueprints.networks.Networks
    variables:
      Networks: 
        InternalComms:
          AssignPublicIp: False
          SecurityGroups: 
            - sg-id
          Subnets: 
            - subnet-id1
            - subnet-id2

and to read that config, i have a blueprint called blueprints/networks.py that has this in it:

class Networks(Blueprint):
"""Manages creation of networks for ECS clusters"""

# Variables that are passed from the my_cluster.yaml
VARIABLES = {
    "Networks": {
        "type": TroposphereType(ecs.AwsvpcConfiguration, many=True),
        "description": "Dictionary for ECS cluster networks"
    }
}

def create_template(self):
    """method that Stacker calls to manipulate the template(s)"""
    variables = self.get_variables()
    for config in variables['Networks']:
        network = ecs.NetworkConfiguration(config)
        t.add_resource(network)
        # do other useful stuff like set outputs

If you're wondering why I create an AwsvpcConfiguration object and then create a NetworkConfiguration object out of that, the reason is because I tried to pass this information in using a NetworkConfiguration object before I adopted the AwsvpcConfiguration object and it didn't work either. I'm using this file to guide me because it's where these objects are defined. The next step is to build the resources, so when I do that by running this command:

stacker build path/to/my.env path/to/my_cluster.yaml

I get an error that says this:

stacker.exceptions.ValidatorError: Validator 'AwsvpcConfiguration.create' 
failed for variable 'Networks' with value 
'{'InternalComms': {'AssignPublicIp': False, 'SecurityGroups': ['sg-id'], 'Subnets': ['subnet-id1', 'subnet-id2']}}': 
TypeError: _from_dict() argument after ** must be a mapping, not str

This is probably my lack of skills at stacker, yaml and python but I'm stumped and have been for a day or so.

I am not able to figure out how to pass the config from the yaml into the blueprint as a dictionary, like I am doing in an identical manner for other resources that do get successfully created in AWS-land. If you could point me to the error(s), I would be much obliged and will for sure tell Santa all about how nice you were.

Solution

  • troposphere maintainer/stacker author here. So a few things:

    1. You want to use TroposphereType(ecs.NetworkConfiguration, many=True) since that's the type of object you want to create.
    2. ecs.NetworkConfiguration (https://github.com/cloudtools/troposphere/blob/master/troposphere/ecs.py#L71) is a AWSProperty type, which means the values inside of it, if using "many" needs to be a list of dictionaries per: http://stacker.readthedocs.io/en/latest/blueprints.html#property-types

    So what I think you want is in your config

    variables: Networks: - AwsvpcConfiguration: AssignPublicIp: False SecurityGroups: - sg-id Subnets: - subnet-id1 - subnet-id2

    This is because TroposphereType expects you to pass in the exact arguments that the type expects. A NetworkConfiguration expects a single key, AwsvpcConfiguration, and the values you were passing were values that were expected by the AwsvcpConfiguration object.

    The bigger question now is how you expect to use those objects. In Cloudformation/troposphere, Property types aren't created on their own - they used as properties of actual resources- in this case the ecs.Service type. Are you not planning to include the Service in the same blueprint? If not, what is your plan to share these properties with other blueprints where those services live?

    A better fit might be to have a blueprint that builds the service along with it's NetworkConfiguration. Then, if you want to have the same NetworkConfiguration easily shared between multiple stacks using that blueprint, you can do something with YAML anchors like:

    common_network_configuration: &common_network_configuration - AwsvpcConfiguration: AssignPublicIp: False SecurityGroups: - sg-id Subnets: - subnet-id1 - subnet-id2

    And then use it wherever you want in variables like:

    variables: << : *common_network_configuration

    I hope that makes sense - if you have any more questions, you can always reach us on the stacker Slack as well: https://empire-slack.herokuapp.com/

    Have a good one!