I am trying to use a subnet (precreated in AWS) in my Blueprint to run it in Apache Brooklyn.
I have the following Blueprint without subnet, which works correctly. (Ambari Server, with Cluster of two Hadoop nodes)
name: IDA Ambari All Services without Security Group
description: This Application is used to build one Ambari Server and a Cluster of Ambari Agents
origin: vendor
location:
jclouds:aws-ec2:
region: eu-central-1
identity: identity
credential: credential
osFamily: RHEL
osVersionRegex: 6.*
hardwareId: m3.medium
services:
- type: io.brooklyn.ambari.AmbariCluster
brooklyn.config:
initialSize: 2
install.version: 2.1.0
services:
- FALCON
And in the moment that I add the new name of the subnet (last lines of the following blueprint), the deployment in Apache Brooklyn fails.
name: IDA Ambari All Services without Security Group
description: This Application is used to build one Ambari Server and a Cluster of Ambari Agents
origin: vendor
location:
jclouds:aws-ec2:
region: eu-central-1
identity: identity
credential: credential
osFamily: RHEL
osVersionRegex: 6.*
hardwareId: m3.medium
services:
- type: io.brooklyn.ambari.AmbariCluster
brooklyn.config:
initialSize: 2
install.version: 2.1.0
services:
- FALCON
provisioning.properties:
networkName: subnet-003dfb7b
I suppose that the problem comes with the configuration of jclouds in the Blueprint. But I do not know.
The errors:
In Nodes
start failed with error: java.lang.IllegalStateException: Node in cluster DynamicClusterImpl{id=tEESwAU9} failed: 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s] Failure running task starting 2 nodes (parallel) (OnIRxglV): 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s]
In the Ambari Server
start failed with error: java.lang.IllegalStateException: Node in cluster DynamicClusterImpl{id=tEESwAU9} failed: 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s] Failure running task starting 2 nodes (parallel) (OnIRxglV): 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s]
Any idea??
The stack trace is a bit dense but in the middle of it we can see:
could not connect to any ip address port 22 on node
This indicates that jclouds is trying to connect to the new node, but it is unable to do so.
The most likely cause is that the instance does not have correctly-configured security groups.
jclouds has a bug which prevents it creating security groups, if a subnet ID is given: see JCLOUDS-987: AWS EC2: if specifying subnet ID, security groups are not created.
You can work around this by creating a security group ahead of time that opens up the required ports - this will be port 22, plus whatever ports are needed by the entity (Ambari in this case). You then pass this to jclouds using Brooklyn's templateOptions property in the blueprint.
For example:
name: IDA Ambari All Services with explicit Security Group
description: This Application is used to build one Ambari Server and a Cluster of Ambari Agents
origin: vendor
location:
jclouds:aws-ec2:
region: eu-central-1
identity: identity
credential: credential
osFamily: RHEL
osVersionRegex: 6.*
hardwareId: m3.medium
services:
- type: io.brooklyn.ambari.AmbariCluster
brooklyn.config:
initialSize: 2
install.version: 2.1.0
services:
- FALCON
provisioning.properties:
networkName: subnet-003dfb7b
templateOptions:
securityGroupIds: sg-12345678