I am using Beautifulsoup with lxml package in my Django app (python 2.7). My production server is OpenShift, so I can not install using apt-get due to access rights. I was able to install lxml to OpenShift in past using easy_install in past, but I struggle to do so now.
Well probably I am missing some dependences that are installed through apt-get like these?
I tried to install lxml by adding lxml to my setup.py file. Did not work.
So I tried pip install lxml==3:
> pip install lxml==3
Downloading/unpacking lxml==3
Downloading lxml-3.0.tar.gz (3.2MB): 3.2MB downloaded
Running setup.py egg_info for package lxml
Building lxml version 3.0.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /usr/lib64
warning: no previously-included files found matching '*.py'
warning: no files found matching '*.txt' under directory 'src/lxml/tests'
Installing collected packages: lxml
Running setup.py install for lxml
Building lxml version 3.0.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /usr/lib64
building 'lxml.etree' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -I/var/lib/openshift/569eccb30c1e661fc20000d5/app-root/runtime/dependencies/python/virtenv/build/lxml/src/lxml/includes -I/opt/rh/python27/root/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o
src/lxml/lxml.etree.c: In function ‘__pyx_pf_4lxml_5etree_4XSLT_18__call__’:
src/lxml/lxml.etree.c:132418: warning: passing argument 1 of ‘__pyx_f_4lxml_5etree_12_XSLTContext__copy’ from incompatible pointer type
src/lxml/lxml.etree.c:130379: note: expected ‘struct __pyx_obj_4lxml_5etree__XSLTContext *’ but argument is of type ‘struct __pyx_obj_4lxml_5etree__BaseContext *’
src/lxml/lxml.etree.c: In function ‘__pyx_f_4lxml_5etree__copyXSLT’:
src/lxml/lxml.etree.c:133807: warning: passing argument 1 of ‘__pyx_f_4lxml_5etree_12_XSLTContext__copy’ from incompatible pointer type
src/lxml/lxml.etree.c:130379: note: expected ‘struct __pyx_obj_4lxml_5etree__XSLTContext *’ but argument is of type ‘struct __pyx_obj_4lxml_5etree__BaseContext *’
src/lxml/lxml.etree.c: At top level:
src/lxml/lxml.etree.c:11938: warning: ‘__pyx_f_4lxml_5etree_displayNode’ defined but not used
Connection to app-nick.rhcloud.com closed by remote host.
Connection to app-nick.rhcloud.com closed.
if I specify subversion (3.4 instead of 3) with pip there is a little bit different output:
pip install lxml==3.4
Downloading/unpacking lxml==3.4
Downloading lxml-3.4.0.tar.gz (3.5MB): 3.5MB downloaded
Running setup.py egg_info for package lxml
/opt/rh/python27/root/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
warnings.warn(msg)
Building lxml version 3.4.0.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /usr/lib64
warning: no previously-included files found matching '*.py'
Installing collected packages: lxml
Running setup.py install for lxml
/opt/rh/python27/root/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
warnings.warn(msg)
Building lxml version 3.4.0.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /usr/lib64
building 'lxml.etree' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -I/var/lib/openshift/569eccb30c1e661fc20000d5/app-root/runtime/dependencies/python/virtenv/build/lxml/src/lxml/includes -I/opt/rh/python27/root/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w
and easy_install:
> easy_install lxml
Searching for lxml
Reading https://pypi.python.org/simple/lxml/
Best match: lxml 3.5.0
Downloading https://pypi.python.org/packages/source/l/lxml/lxml-3.5.0.tar.gz#md5=9f0c5f1eb43ff44d5455dab4b4efbe73
Processing lxml-3.5.0.tar.gz
Writing /tmp/easy_install-K7xT2d/lxml-3.5.0/setup.cfg
Running lxml-3.5.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-K7xT2d/lxml-3.5.0/egg-dist-tmp-BAe44x
Building lxml version 3.5.0.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /usr/lib64
Connection to app-nick.rhcloud.com closed by remote host.
Connection to app-nick.rhcloud.com closed.
How do I install lxml through pip or easy_install only? Or should I use another xml parser with BeautifulSoup (not sure if there are alternatives)?
UPDATE1
I tried to install lxml==3.2.5. I forgot to mention I am on python 2.7 gear. If I add , 'lxml==3.2.5' to my setup.py file(I think setup.py works the same as requirements.txt), this is output:
remote: Searching for lxml==3.2.5
remote: Reading http://mirror1.ops.rhcloud.com/mirror/python/web/simple/lxml/
remote: Best match: lxml 3.2.5
remote: Downloading http://mirror1.ops.rhcloud.com/mirror/python/web/packages/source/l/lxml/lxml-3.2.5.tar.gz#md5=6c4fb9b1840631cff09b8229a12a9ef7
remote: Processing lxml-3.2.5.tar.gz
remote: Writing /tmp/easy_install-jvJHz8/lxml-3.2.5/setup.cfg
remote: Running lxml-3.2.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-jvJHz8/lxml-3.2.5/egg-dist-tmp-KMpa_f
remote: /opt/rh/python27/root/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
remote: warnings.warn(msg)
remote: warning: no files found matching '*.txt' under directory 'src/lxml/tests'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_4data':
remote: src/lxml/lxml.etree.c:97814: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxData' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97415: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_6start':
remote: src/lxml/lxml.etree.c:97951: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxStart' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97115: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_8end':
remote: src/lxml/lxml.etree.c:98009: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxEnd' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97339: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_10pi':
remote: src/lxml/lxml.etree.c:98167: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxPi' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97455: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_12comment':
remote: src/lxml/lxml.etree.c:98220: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxComment' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97581: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__xslt_resolve_from_python':
remote: src/lxml/lxml.etree.c:136117: warning: enumeration value '__pyx_e_4lxml_5etree_PARSER_DATA_INVALID' not handled in switch
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_4XSLT_18__call__':
remote: src/lxml/lxml.etree.c:140191: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type
remote: src/lxml/lxml.etree.c:138126: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *'
remote: src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__copyXSLT':
remote: src/lxml/lxml.etree.c:141589: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type
remote: src/lxml/lxml.etree.c:138126: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *'
Connection to app-host.rhcloud.com closed by remote host.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
UPDATE2
I tried to install Cython (pip install cython), it also failed -> I was disconnected from OpenShift.
I have created bug report in RedHat's bugzilla.
Weird bug, Maciej from RedHat suggested I recreate Openshift app again from scratch. It worked and I was able to install lxml successfully.