I run a vnc server on my workstation and connect with another computer. Both of the server and client are running Debian Jessie:
$ uname -a
Linux debian-VAIO 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u2 (2016-06-25) x86_64 GNU/Linux
Xfce is installed for the vnc server and xstartup is:
$ cat ~/.vnc/xstartup
#!/bin/sh
xrdb $HOME/.Xresources
xsetroot -solid grey
export XKL_XMODMAP_DISABLE=1
exec startxfce4
In the ~/.bashrc, I have added the lines below:
export LIBGL_ALWAYS_INDIRECT=y
export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libstdc++.so.6'
Currently, when I ssh to the server with ssh -X, the $DISPLAY is localhost:10.0, and the vtk related command will launch an X window. However, if a vnc server is launched in the ssh terminal, logging into xfce and opening a terminal in it, the $DISPLAY is :1.0 and vtk fails to launch an X window, with the following message:
Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 394
vtkXOpenGLRenderWindow (0x2c30f10): Could not find a decent visual
Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 394
vtkXOpenGLRenderWindow (0x2c30f10): Could not find a decent visual
Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 394
vtkXOpenGLRenderWindow (0x2c30f10): Could not find a decent visual
Error: In /home/orobix/Desktop/vmtk-build/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx, line 613
vtkXOpenGLRenderWindow (0x2c30f10): GLX not found. Aborting.
I think it might be relate to the missed config of X server in xstartup of vnc server. But I don't know how to do it. Could anyone help me debug it? Any further information will be provided if needed. Thanks!
20160823 Update
I accepted the suggestion of VirtualGL + TurboVNC and installed the two components. A simple config using vglserver_config was done according to http://www.virtualgl.org/vgldoc/2_1_1/#hd009001 part 6.1. Then I made vncserver of TurboVNC to run xfce with xstartup.turbovnc:
#!/bin/sh
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
OS=`uname -s`
exec startxfce4
On the client, I use vncviewer of TurboVNC to connect the server. The desktop looks quite different from that in a default vncserver. Then I tried
/opt/VirtualGL/bin/vglrun vmtkimageviewer -ifile image_volume_voi.vti
in which the vmtkimageviewer should open up a window, but finally it gives out an error:
Executing vmtkimageviewer ...
X Error of failed request: GLXBadContext
Major opcode of failed request: 156 (GLX)
Minor opcode of failed request: 6 (X_GLXIsDirect)
Serial number of failed request: 17
Current serial number in output stream: 16
And unfortunately, I find that with ssh -X, I cannot launch the X window as I did before, although $DISPLAY is still localhost:10.0. The output is:
Executing vmtkimageviewer ...
X Error of failed request: BadValue (integer parameter out of range for operation)
Major opcode of failed request: 156 (GLX)
Minor opcode of failed request: 3 (X_GLXCreateContext)
Value in failed request: 0x0
Serial number of failed request: 37
Current serial number in output stream: 38
But I think I am closer to the fact, as it looks like VirtualGL works(?). What to do for the next?
Plain and simple the X server variant that's being used by Xrdp (either Xvnc or Xvfb) simply doesn't support OpenGL / GLX and thus programs that need OpenGL will not work in that configuration.
Fallback to Mesa swrast or llvmpipe is possible though: http://www.mesa3d.org/llvmpipe.html