We were running pyodbc 4.0.1 in our Production environment and we suddenly started to face memory errors. I'm not sure why it started to happen since everything was working fine until then... Is there any known root cause for the issue in terms of server configuration?
We have upgraded our version of Pyodbc from 4.0.1 to 4.0.16 in DEV and UAT. Unfortunately the issue is still there and reproducible (see trace below).
[...]
*** glibc detected *** python: corrupted double-linked list: 0x0000000002644650 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75f4e)[0x7f751c639f4e]
/lib64/libc.so.6(+0x79066)[0x7f751c63d066]
/usr/lib64/libtdsodbc.so.0(+0x1e2fd)[0x7f74fd2aa2fd]
/usr/lib64/libtdsodbc.so.0(+0x2095e)[0x7f74fd2ac95e]
/usr/lib64/libtdsodbc.so.0(+0x27de5)[0x7f74fd2b3de5]
/usr/lib64/libtdsodbc.so.0(+0x3729f)[0x7f74fd2c329f]
/usr/lib64/libtdsodbc.so.0(+0x11e15)[0x7f74fd29de15]
/usr/lib64/libtdsodbc.so.0(SQLGetTypeInfo+0x187)[0x7f74fd2a0547]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/lib/libodbc.so.2(SQLGetTypeInfo+0x1e3)[0x7f75033c8c93]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/lib/python3.5/site-packages/pyodbc.cpython-35m-x86_64-linux-gnu.so(_Z17GetConnectionInfoP7_objectP10Connection+0x42f)[0x7f75036297af]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/lib/python3.5/site-packages/pyodbc.cpython-35m-x86_64-linux-gnu.so(_Z14Connection_NewP7_objectbblbS0_R6Object+0x6a9)[0x7f7503621249]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/lib/python3.5/site-packages/pyodbc.cpython-35m-x86_64-linux-gnu.so(+0xb174)[0x7f7503622174]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyCFunction_Call+0x131)[0x7f751d4c4621]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x9017)[0x7f751d54bc37]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x9546)[0x7f751d54c166]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x9546)[0x7f751d54c166]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x9546)[0x7f751d54c166]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(+0x144b49)[0x7f751d54cb49]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x48)[0x7f751d54ccd8]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyEval_EvalCode+0x3b)[0x7f751d54cd1b]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyRun_FileExFlags+0x130)[0x7f751d572020]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(PyRun_SimpleFileExFlags+0x173)[0x7f751d573623]
/opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/../lib/libpython3.5m.so.1.0(Py_Main+0xca7)[0x7f751d58e8c7]
python(main+0x15d)[0x400add]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f751c5e2d5d]
python[0x4008b9]
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:05 11011012 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/python3.5
00601000-00602000 rw-p 00001000 fd:05 11011012 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/envs/ed/bin/python3.5
01241000-02873000 rw-p 00000000 00:00 0 [heap]
7f74f8000000-7f74f8021000 rw-p 00000000 00:00 0
7f74f8021000-7f74fc000000 ---p 00000000 00:00 0
7f74fc354000-7f74fc356000 r-xp 00000000 fd:01 3837 /usr/lib64/gconv/CP1252.so
7f74fc356000-7f74fc555000 ---p 00002000 fd:01 3837 /usr/lib64/gconv/CP1252.so
7f74fc555000-7f74fc556000 r--p 00001000 fd:01 3837 /usr/lib64/gconv/CP1252.so
7f74fc556000-7f74fc557000 rw-p 00002000 fd:01 3837 /usr/lib64/gconv/CP1252.so
7f74fc557000-7f74fc558000 r-xp 00000000 fd:01 4029 /usr/lib64/gconv/ISO8859-1.so
7f74fc558000-7f74fc758000 ---p 00001000 fd:01 4029 /usr/lib64/gconv/ISO8859-1.so
7f74fc758000-7f74fc759000 r--p 00001000 fd:01 4029 /usr/lib64/gconv/ISO8859-1.so
7f74fc759000-7f74fc75a000 rw-p 00002000 fd:01 4029 /usr/lib64/gconv/ISO8859-1.so
7f74fc75a000-7f74fc75f000 r-xp 00000000 fd:01 3803 /lib64/libnss_dns-2.12.so
7f74fc75f000-7f74fc95e000 ---p 00005000 fd:01 3803 /lib64/libnss_dns-2.12.so
7f74fc95e000-7f74fc95f000 r--p 00004000 fd:01 3803 /lib64/libnss_dns-2.12.so
7f74fc95f000-7f74fc960000 rw-p 00005000 fd:01 3803 /lib64/libnss_dns-2.12.so
7f74fc960000-7f74fc963000 r-xp 00000000 fd:01 5388 /lib64/libgpg-error.so.0.5.0
7f74fc963000-7f74fcb62000 ---p 00003000 fd:01 5388 /lib64/libgpg-error.so.0.5.0
7f74fcb62000-7f74fcb63000 r--p 00002000 fd:01 5388 /lib64/libgpg-error.so.0.5.0
7f74fcb63000-7f74fcb64000 rw-p 00003000 fd:01 5388 /lib64/libgpg-error.so.0.5.0
7f74fcb64000-7f74fcb74000 r-xp 00000000 fd:01 6697 /usr/lib64/libtasn1.so.3.1.6
7f74fcb74000-7f74fcd73000 ---p 00010000 fd:01 6697 /usr/lib64/libtasn1.so.3.1.6
7f74fcd73000-7f74fcd74000 rw-p 0000f000 fd:01 6697 /usr/lib64/libtasn1.so.3.1.6
7f74fcd74000-7f74fcde6000 r-xp 00000000 fd:01 5401 /lib64/libgcrypt.so.11.5.3
7f74fcde6000-7f74fcfe5000 ---p 00072000 fd:01 5401 /lib64/libgcrypt.so.11.5.3
7f74fcfe5000-7f74fcfe6000 r--p 00071000 fd:01 5401 /lib64/libgcrypt.so.11.5.3
7f74fcfe6000-7f74fcfe9000 rw-p 00072000 fd:01 5401 /lib64/libgcrypt.so.11.5.3
7f74fcfe9000-7f74fd086000 r-xp 00000000 fd:01 13301 /usr/lib64/libgnutls.so.26.14.12
7f74fd086000-7f74fd285000 ---p 0009d000 fd:01 13301 /usr/lib64/libgnutls.so.26.14.12
7f74fd285000-7f74fd28c000 rw-p 0009c000 fd:01 13301 /usr/lib64/libgnutls.so.26.14.12
7f74fd28c000-7f74fd2ee000 r-xp 00000000 fd:01 52715 /usr/lib64/libtdsodbc.so.0.0.0
7f74fd2ee000-7f74fd4ed000 ---p 00062000 fd:01 52715 /usr/lib64/libtdsodbc.so.0.0.0
7f74fd4ed000-7f74fd4f1000 rw-p 00061000 fd:01 52715 /usr/lib64/libtdsodbc.so.0.0.0
7f74fd4f1000-7f74fd4f5000 r-xp 00000000 fd:05 7997029 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/termios.so
7f74fd4f5000-7f74fd6f5000 ---p 00004000 fd:05 7997029 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/termios.so
7f74fd6f5000-7f74fd6f7000 rw-p 00004000 fd:05 7997029 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/termios.so
7f74fd6f7000-7f74fd6fa000 r-xp 00000000 fd:05 7997070 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/fcntl.so
7f74fd6fa000-7f74fd8f9000 ---p 00003000 fd:05 7997070 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/fcntl.so
7f74fd8f9000-7f74fd8fb000 rw-p 00002000 fd:05 7997070 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/python-3.5.2-0/lib/python3.5/lib-dynload/fcntl.so
7f74fd8fb000-7f74fd8fe000 r-xp 00000000 fd:05 10749483 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cresultproxy.so
7f74fd8fe000-7f74fdafe000 ---p 00003000 fd:05 10749483 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cresultproxy.so
7f74fdafe000-7f74fdaff000 rw-p 00003000 fd:05 10749483 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cresultproxy.so
7f74fdaff000-7f74fdb00000 r-xp 00000000 fd:05 10749167 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cutils.so
7f74fdb00000-7f74fdd00000 ---p 00001000 fd:05 10749167 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cutils.so
7f74fdd00000-7f74fdd01000 rw-p 00001000 fd:05 10749167 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cutils.so
7f74fdd01000-7f74fdd41000 rw-p 00000000 00:00 0
7f74fdec1000-7f74fdf01000 rw-p 00000000 00:00 0
7f74fdf41000-7f74fdf44000 r-xp 00000000 fd:05 10749471 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cprocessors.so
7f74fdf44000-7f74fe143000 ---p 00003000 fd:05 10749471 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cprocessors.so
7f74fe143000-7f74fe144000 rw-p 00002000 fd:05 10749471 /opt/cloudera/parcels/Anaconda3-4.2.0-python-3.5/pkgs/sqlalchemy-1.0.13-py35_0/lib/python3.5/site-packages/sqlalchemy/cprocessors.soAborted
After doing some more tests on our ends, it seems the issue is occurring when connecting to Microsoft SQL Server databases. We connect dsn-less with the following connection string. This being said, I'm not sure when is the memory issue occurring (before / during / after the connection)... The issue seems to be triggered more or less randomly.
[...]
pyodbc.pooling=False
oConnexion = pyodbc.connect(Driver="{FreeTDS};Server=xxx;Port=1433;TDS_Version=7.0;UID=xxx;PWD=xxx;", autocommit=True)
oConnexion.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
oConnexion.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-8')
oConnexion.setdecoding(pyodbc.SQL_WMETADATA, encoding='utf-8')
oConnexion.setencoding(encoding='utf-8')
[...]
Configuration:
- Python 3.5.2
- Anaconda 4.2.0 (64-bit)
- Red Hat 4.4.7-1 on linux
- Pyodbc 4.0.16
Previous configuration of the ODBC Driver was FreeTDS v0.91. Upgrading ODBC Driver to FreeTDS 1.0rc5 solved the issue.