x11 forwarding with paramiko

I'm trying to run a command with paramiko that should be able to open an X window. The script I'm using would something as follows:

import paramiko                                    

ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect('192.168.122.55', username='user', password='password')
transport = ssh_client.get_transport()
session = transport.open_session()

session.request_x11()
stdin = session.makefile('wb')
stdout = session.makefile('rb')
stderr = session.makefile_stderr('rb')
session.exec_command('env; xterm')
transport.accept()

print 'Exit status:', session.recv_exit_status()
print 'stdout:\n{}'.format(stdout.read())
print 'stderr:\n{}'.format(stderr.read())
session.close()

Unfortunately, when I run the script above, I get this output:

Exit status: 1
stdout:
SHELL=/bin/bash
XDG_SESSION_COOKIE=8025e1ba5e6c47be0d2f3ad6504a25ee-1347286654.617967-1932974971
SSH_CLIENT=192.168.122.1 58654 22
USER=user
MAIL=/var/mail/user
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PWD=/home/user
LANG=en_US.UTF-8
SHLVL=1
HOME=/home/user
LOGNAME=user
SSH_CONNECTION=192.168.122.1 58654 192.168.122.55 22
DISPLAY=localhost:10.0
_=/usr/bin/env

stderr:  
xterm: Xt error: Can't open display: localhost:10.0

If I run the following command in a terminal:

ssh -X user@192.168.122.55 'env; xterm'

then I get the same environment variables (some ports changed, though), so I'd say that my environment is correct. However, I'm still missing something to make paramiko work with x11 forwarding.

A couple of things I tried are:

Use the handler parameter in request_x11: aside from printing values, I didn't get any further than with the default handler.
Use the auth_cookie parameter in request_x11: tried to hardcode a cookie value that was being used according to the xauth list output. The idea of doing this was to avoid problems that might happen according to the documentation string in paramiko itself:

If you omit the auth_cookie, a new secure random 128-bit value will be generated, used, and returned. You will need to use this value to verify incoming x11 requests and replace them with the actual local x11 cookie (which requires some knoweldge of the x11 protocol).

Is there some other thing I could do to make it work or troubleshoot the problem?

Note: This has been previously asked in:

superuser: the only response points to the request_x11 documentation I've already tried to use to no avail.
stackoverflow: the accepted response suggests to use the handler parameter, but it's wrong.
github: no answer provided for more than a year.

Solution

the x11 request may use a MIT-MAGIC-COOKIE-1 that you may not handled properly
using ssh directly I saw it needed to confirm the x11 request (cookie challenge?)
the .Xauthority file may also be an issue
you can try to strace ssh process and see a normal flow
in your script, you can replace xterm with strace xterm and compare with the above.

some links:

good luck.

EDIT: building on top of Gary's answer, with multiple x11 connections.

#!/usr/bin/env python

import os
import select
import sys
import getpass
import paramiko
import socket
import logging
import Xlib.support.connect as xlib_connect
LOGGER = logging.getLogger(__name__)

# connection settings
host = '192.168.122.55'
user = 'user'
password = getpass.getpass()

ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(host, username=user, password=password)
del password

# maintain map
# { fd: (channel, remote channel), ... }
channels = {}

poller = select.poll()
def x11_handler(channel, (src_addr, src_port)):
    '''handler for incoming x11 connections
    for each x11 incoming connection,
    - get a connection to the local display
    - maintain bidirectional map of remote x11 channel to local x11 channel
    - add the descriptors to the poller
    - queue the channel (use transport.accept())'''
    x11_chanfd = channel.fileno()
    local_x11_socket = xlib_connect.get_socket(*local_x11_display[:3])
    local_x11_socket_fileno = local_x11_socket.fileno()
    channels[x11_chanfd] = channel, local_x11_socket
    channels[local_x11_socket_fileno] = local_x11_socket, channel
    poller.register(x11_chanfd, select.POLLIN)
    poller.register(local_x11_socket, select.POLLIN)
    LOGGER.debug('x11 channel on: %s %s', src_addr, src_port)
    transport._queue_incoming_channel(channel)

def flush_out(session):
    while session.recv_ready():
        sys.stdout.write(session.recv(4096))
    while session.recv_stderr_ready():
        sys.stderr.write(session.recv_stderr(4096))

# get local disply
local_x11_display = xlib_connect.get_display(os.environ['DISPLAY'])
# start x11 session
transport = ssh_client.get_transport()
session = transport.open_session()
session.request_x11(handler=x11_handler)
session.exec_command('xterm')
session_fileno = session.fileno()
poller.register(session_fileno, select.POLLIN)
# accept first remote x11 connection
transport.accept()

# event loop
while not session.exit_status_ready():
    poll = poller.poll()
    # accept subsequent x11 connections if any
    if len(transport.server_accepts) > 0:
        transport.accept()
    if not poll: # this should not happen, as we don't have a timeout.
        break
    for fd, event in poll:
        if fd == session_fileno:
            flush_out(session)
        # data either on local/remote x11 socket
        if fd in channels.keys():
            channel, counterpart = channels[fd]
            try:
                # forward data between local/remote x11 socket.
                data = channel.recv(4096)
                counterpart.sendall(data)
            except socket.error:
                channel.close()
                counterpart.close()
                del channels[fd]

print 'Exit status:', session.recv_exit_status()
flush_out(session)
session.close()