AWS/Ansible - How to access facts across roles/hosts from host defined in dynamic-inventory?

Question

I am currently setting up a number of Ansible roles to setup a Kubernetes cluster. So far I have a role to provision idempotent EC2s (1x Master / 2x Worker) and subsequent roles to setup these master/worker nodes with Docker/Kubernetes dependencies. I am using AWS ec2.ini/.py dynamic-inventory to discover IPs of the instances provisioned by my create_ec2 role.

I have encountered an issue when trying to join my workers to the cluster with the join command I am retrieving from the master node. I have 2 seperate roles for the master & worker provisioning. In the tasks for the master, I get the join command with:

kubeadm token create --print-join-command

and then register a variable which I then use to set a host fact:

set_fact:
  join_command: "{{ join_command_stdout.stdout_lines[0] }}"

The issue I am having is when I try to access this fact on my worker nodes when running my worker role. I am trying to access the fact with:

"{{ hostvars['tag_Type_master'].join_command }} --ignore-preflight-errors all  >> node_joined.txt"

However it is failing as the host I am providing for the hostvars is apparently undefined..

For reference, I have this value held in my dynamic-inventory (IP omitted):

  "tag_Type_master": [
    "1.2.3.4"

The error I am receiving is:

"{"msg": "The task includes an option with an undefined variable. The error was: \"hostvars['tag_Type_master']\" is undefined"

I am struggling to figure out how I access the host facts of an EC2 instance defined in my dynamic-inventory.

I have tried supplementing the EC2 IP directly into the hostvars (hostvars['1.2.3.4'].join_command), however the task just hangs and does nothing.

I have also tried putting in a Magic variable (hostvars['inventory_hostname].join_command) to no avail.

It seems that people have had success with accessing host facts from hosts defined in a static inventory file, however due to the dynamic nature of the EC2 servers the cluster will be created on I am unable to use this approach.

run.yml:

  name: Setup K8s master node
  hosts: tag_Name_kube_master    
  gather_facts: true    
  roles:    
  - setup_kube_master


  name: Setup K8s worker nodes
  hosts: tag_Name_kube_worker
  gather_facts: true
  roles: 
  - setup_kube_worker

setup_kube_master/tasks/set_fact.yml:

  name: Get join command for workers    
  shell: kubeadm token create --print-join-command    
  register: join_command_stdout    
  name: Persist variable for workers    
  set_fact:    
   join_command: "{{ join_command_stdout.stdout_lines[0] }}"

setup_kube_worker/tasks/get_fact.yml:

  name: join cluster
  shell: "{{ hostvars['tag_Type_master'].join_command }} --ignore-preflight-errors all  >> node_joined.txt"    
  args:    
   chdir: $HOME    
   creates: node_joined.txt

Answer 1

So the way you would troubleshoot this for yourself is to use the debug: task to show the entire fact cache and find the relationship for yourself:

- name: show the state of affairs
  debug: var=hostvars verbosity=0

However, having said that, I'm pretty sure that tag_Type_master is defined as a group and thus will not show up in hostvars since -- as its name implies -- it is vars for hosts not vars for groups

You have to do one level of indirection to obtain a host that is a member of that group:

- hosts: tag_Name_kube_worker
  tasks:
  - name: promote the "join_command" fact from the masters group
    set_fact:
      join_command: '{{ some_master.join_command }}'
    vars:
      some_master: '{{ hostvars[groups["tag_Type_master"][0]] }}'

I took some liberties with that some_master definition for the sake of brevity -- in production code you would want to actually check that that group exists and its contents are not empty, etc, etc, but I'm about 80% sure it will work even as written

You would want that to appear in run.yml in between the hosts: tag_Type_master and hosts: tag_Type_worker to bridge the fact-gap between the two groups and make it appear as if the workers had that join_command fact the whole time

---

Separately, while this isn't what you asked, if you were to tag those instances with "kubernetes.io/role/master": "" and/or "kubernetes.io/role": "master" you would benefit by already having the tags that the cloud-provider is expecting. I have no idea what that would look like in ec2.py, but I'm sure it would be cheap to find out using ansible-inventory -i ec2.py --list

I tag the workers with the corresponding kubernetes.io/role: worker even though I'm pretty sure the AWS cloud-provider doesn't care about it, choosing instead to just use the metadata.labels on the existing Nodes for doing ELB registration et al.