Obtain lish console to my linode with ansible and issue commands

Question

I'm playing with ansible and my pet linode and got into a rabbit hole.

Here is current version of my playbook:

- name: run command through Linode Lish
  hosts: localhost
  gather_facts: false

  vars_files:
    - ../group_vars/vars.yml

  tasks:
    - include_tasks: ../tasks/get_console_through_lish_and_run_commands.yml
      # https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html
      vars:
        ansible_connection: ansible.builtin.ssh
        ansible_ssh_host: lish-ap-southeast.linode.com
        ansible_ssh_user: my_user_name
        ansible_ssh_common_args: "-t -o ControlMaster=auto -o ControlPersist=300s"
        ansible_ssh_timeout: 300
        connect_timeout: 30

Here is task:

- name: connect console through lish
  ansible.builtin.raw: my_linode_name
  register: ssh_command_output

- name: Print response of ssh command
  debug:
    msg: "{{ ssh_command_output }}"


- name: run command through lish console
  ansible.builtin.raw: ls -lah
  register: ssh_command_output

- name: Print response of ssh command
  debug:
    msg: "{{ ssh_command_output }}"

The first task is always hanging. When I invoked playbook with ANSIBLE_DEBUG=1 I see that task gets the linode console but then it stays there for ever. It seems obtaining console in lish does not provide feedback required by ansible for ansible to know that the task completed with success (in this case that I got console to my linode)

I was naively thinking I can issue my linode name in one task, let it timeout and then have second task pick up the ssh connection with the console ready for commands.

So I added following to the first task:

- name: connect console through lish
  ansible.builtin.raw: my_linode_name
  register: ssh_command_output
  timeout: 20
  ignore_errors: true

But it seems ssh connections is reset when the task timeouts.

I'm out of ideas what else to try to get ansible to issue commands through linode lish console, please hint me if there are other ways to achieve this. Many thanks in advance!

---edit 2 days later---

I have some progress but still wondering in the dark.

If I label my linode with prefix, say, linode_ then I can add something like below to ~/.ssh/config

Match exec "echo %h | grep -E 'linode_'"
    HostName lish-ap-southeast.linode.com
    Port 22
    IdentityFile ~/.ssh/id_rsa
    User my_user_name
    RequestTTY yes
    RemoteCommand %n

This lets me to issue ssh linode_whatever and it automatically gets me into my linode.

Sadly, I can't do something like ssh linode_whatever ls -lah which I think is required by ansible...

I also tried to open a tunnel ssh -L 2222:localhost:22 linode_whatever, however as soon as I try ssh -p 2222 localhost I get error channel 2: open failed: administratively prohibited: open failed

---edit 2 days later---

After viewing this answer I thought about starting ssh in the background and sending commands through named pipe

fifo_path=$(mktemp)
rm "${fifo_path}"
mkfifo "${fifo_path}"
cat > "${fifo_path}" &
ssh -t [email protected] > /tmp/logs.txt 2>&1 < "${fifo_path}" &
echo my_linode_name > "${fifo_path}"

on another terminal:

tail -f /tmp/logs.txt

I can issue command like list, however as soon as I issue name of linode to screen to I see an error No pseudo-tty detected.

---edit---

I finally see light at the end of the tunnel.

Following allows me to start ssh to linode lish and get screen sessison to my host which I can control over named pipe:

screen -dm -S workspace /bin/bash -i
screen -S workspace -X stuff 'eval $(ssh-agent -s)\n'
screen -S workspace -X stuff 'ssh-keyscan lish-ap-southeast.linode.com >> ~/.ssh/known_hosts\n'
fifo_path=$(mktemp)
rm "${fifo_path}"
mkfifo "${fifo_path}"
cat > "${fifo_path}" &
screen -S workspace -X stuff 'ssh -t -t [email protected] > /tmp/logs.txt 2>&1 < "'"${fifo_path}"'"\n'
echo "gv_gitlab" > "${fifo_path}"
echo "ls -lah" > "${fifo_path}"

In another terminal:

tail -f /tmp/logs.txt

I can see successfully executed ls -lah within lish console of my linode!

Answer 1

Luckily my rabbit hole had an exit, I'm able to issue commands through linode lish using ansible! Here is a task that demonstrates the idea (it's very rough, the purpose is to show that it's possible to do what I wanted to do):

- name: issue command to lish console
  ansible.builtin.shell: |
    is_pattern_in_log_last_line() { grep -q "$1" <<< "$(tail -n 1 /tmp/logs.txt | tr -d '\r' | tr -d '\n')"; }
    wait_for_pattern() { for i in `seq 1 30`; do echo "($i) Waiting for '$1'"; sleep 1; is_pattern_in_log_last_line "$1" && break; done; is_pattern_in_log_last_line "$1" || exit 1; }
    fifo_path=$(mktemp)
    rm "${fifo_path}"
    mkfifo "${fifo_path}"
    screen -dm -S workspace /bin/bash -i
    screen -S workspace -X stuff 'eval $(ssh-agent -s)\n'
    screen -S workspace -X stuff 'cat > "'"${fifo_path}"'" &\n'
    screen -S workspace -X stuff 'ssh-keyscan lish-ap-southeast.linode.com >> ~/.ssh/known_hosts\n'
    screen -S workspace -X stuff 'ssh -t -t [email protected] > /tmp/logs.txt 2>&1 < "'"${fifo_path}"'"\n'
    wait_for_pattern "\[[email protected]\]#"
    echo "my_linode_name" > "${fifo_path}"
    wait_for_pattern "root@finnix:~#"
    echo "ls -lah" > "${fifo_path}"
    sleep 5
    screen -X -S workspace quit
    cat /tmp/logs.txt
  args:
    executable: /bin/bash
  delegate_to: localhost
  register: ssh_command_output

- name: Print result
  debug:
    msg: "{{ ssh_command_output }}"

The task assumes ssh won't ask for any input, so you need to setup keys in your linode console and also add them to lish.