.net-coreodbcignite
Apache Ignite 2.7: ODBC Linux - Bad data returned
I am testing out the Apache Ignite 2.7 Linux ODBC Driver, and it's not working correctly for some reason.
When I run a SQL statement, like SELECT Id FROM MyTable, the value is "garbled" returning a value like 8\00\05\07\09\0d\09\08\0-\0 for the the Id column. It's similar to the value I expect, though, which is 80579d98-9010-4610-b12e-ed33ed7d3c62. I don't know if those are ASCII null characters (\0), or what, but something isn't right and I can't figure it out.
When I use the SQLLine tool, the values are not "garbled". I also installed the Windows ODBC driver, and that worked fine, but I need the Linux ODBC Driver.
I built the driver with the following libraries:
gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
GNU Make 4.1
libtoolize 2.4.6
aclocal 1.15
autoheader 2.69
automake 1.15
autoreconf 2.69
unixodbc 2.3.4-1
libssl 1.0.2r-1~deb9u1
Here's the contents of the ignite-odbc-install.ini:
# cat /src/apache-ignite-2.7.0-bin/platforms/cpp/odbc/install/ignite-odbc-install.ini
[Apache Ignite]
Description=Apache Ignite
Driver=/usr/local/lib/libignite-odbc.so
Setup=/usr/local/lib/libignite-odbc.so
DriverODBCVer=03.00
FileUsage=0
Here's the result of running odbcinst:
# odbcinst -i -d -f /src/apache-ignite-2.7.0-bin/platforms/cpp/odbc/install/ignite-odbc-install.ini
odbcinst: Driver installed. Usage count increased to 1.
Target directory is /etc
Here's the contents of the /usr/local/lib folder:
# cd /usr/local/lib
# ls
libignite-binary-2.7.0.33575.so.0 libignite-common-2.7.0.33575.so.0 libignite-odbc-2.7.0.33575.so.0 libignite-thin-client-2.7.0.33575.so.0
libignite-binary-2.7.0.33575.so.0.0.0 libignite-common-2.7.0.33575.so.0.0.0 libignite-odbc-2.7.0.33575.so.0.0.0 libignite-thin-client-2.7.0.33575.so.0.0.0
libignite-binary.a libignite-common.a libignite-odbc.a libignite-thin-client.a
libignite-binary.la libignite-common.la libignite-odbc.la libignite-thin-client.la
libignite-binary.so libignite-common.so libignite-odbc.so libignite-thin-client.so
Here's the result from the ldd command:
# ldd /usr/local/lib/libignite-odbc.so
linux-vdso.so.1 (0x00007fffa46d0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f18d554c000)
libignite-binary-2.7.0.33575.so.0 => /usr/local/lib/libignite-binary-2.7.0.33575.so.0 (0x00007f18d531d000)
libodbcinst.so.2 => /usr/lib/x86_64-linux-gnu/libodbcinst.so.2 (0x00007f18d5108000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f18d4d86000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f18d4a82000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f18d46e3000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f18d44cc000)
/lib64/ld-linux-x86-64.so.2 (0x00007f18d59f8000)
libignite-common-2.7.0.33575.so.0 => /usr/local/lib/libignite-common-2.7.0.33575.so.0 (0x00007f18d42b5000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f18d40ab000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f18d3e8e000)
UPDATE 1
I ran this from a .Net Core 2.2 C# Console program, with only the System.Data.Odbc 4.5.0 NuGet reference in a Docker Linux Environment.
IgniteGarbledDataReproducer.csproj
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.2</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="System.Data.Odbc" Version="4.5.0" />
</ItemGroup>
</Project>
Program.cs
using System;
using System.Data.Odbc;
namespace IgniteGarbledDataReproducer {
internal class Program {
private static void Main(string[] args) {
Console.WriteLine("START");
var connectionString = Environment.GetEnvironmentVariable("IGNITE_CONNECTION_STRING");
Console.WriteLine(connectionString);
using (var conn = new OdbcConnection(connectionString)) {
conn.Open();
Console.WriteLine("Connection Opened");
using (var cmd = conn.CreateCommand()) {
cmd.CommandText = "DROP TABLE IF EXISTS MyTable;";
cmd.ExecuteNonQuery();
}
Console.WriteLine("Table Dropped (if it existed)");
using (var cmd = conn.CreateCommand()) {
cmd.CommandText = "CREATE TABLE MyTable (Id varchar(36) not null, FirstName varchar(255), LastName varchar(255), PRIMARY KEY (Id));";
cmd.ExecuteNonQuery();
}
Console.WriteLine("Table Created");
using (var cmd = conn.CreateCommand()) {
cmd.CommandText = "INSERT INTO MyTable (Id) VALUES ('80579d98-9010-4610-b12e-ed33ed7d3c62');";
cmd.ExecuteNonQuery();
}
Console.WriteLine("Data Inserted");
using (var cmd = conn.CreateCommand()) {
cmd.CommandText = "SELECT Id FROM MyTable;";
using (var rdr = cmd.ExecuteReader()) {
while (rdr.Read())
Console.WriteLine($"Id = '{rdr["Id"]}'");
}
}
Console.WriteLine("Data Returned");
}
Console.WriteLine("DONE");
}
}
}
Dockerfile
FROM microsoft/dotnet:2.2-runtime AS base
WORKDIR /app
WORKDIR /src
# Install pre-requisites for building and installing Ignite ODBC Driver
RUN apt-get update \
&& apt-get install -y gcc g++ make automake autotools-dev libtool m4 libssl1.0-dev unixodbc-dev unixodbc-bin unzip \
&& apt-get clean -y
# Download and unzip Ignite binary release
#ADD http://apache.mirrors.hoobly.com/ignite/2.7.0/apache-ignite-2.7.0-bin.zip /src/
COPY apache-ignite-2.7.0-bin.zip .
RUN unzip -q /src/apache-ignite-2.7.0-bin.zip \
&& rm /src/apache-ignite-2.7.0-bin.zip
# Build Ignite ODBC Driver
ARG IGNITE_HOME=/src/apache-ignite-2.7.0-bin
WORKDIR /src/apache-ignite-2.7.0-bin/platforms/cpp
RUN libtoolize \
&& aclocal \
&& autoheader \
&& automake --add-missing \
&& autoreconf \
&& ./configure --enable-odbc --disable-core --disable-node \
&& make \
&& make install
# Install Ignite ODBC Driver
RUN apt-get update \
&& apt-get install -y unixodbc \
&& apt-get clean -y \
&& odbcinst -i -d -f /src/apache-ignite-2.7.0-bin/platforms/cpp/odbc/install/ignite-odbc-install.ini
FROM microsoft/dotnet:2.2-sdk AS build
WORKDIR /src
COPY IgniteGarbledDataReproducer/IgniteGarbledDataReproducer.csproj IgniteGarbledDataReproducer/
RUN dotnet restore IgniteGarbledDataReproducer/IgniteGarbledDataReproducer.csproj
COPY . .
WORKDIR /src/IgniteGarbledDataReproducer
RUN dotnet build IgniteGarbledDataReproducer.csproj -c Release -o /app
FROM build AS publish
RUN dotnet publish IgniteGarbledDataReproducer.csproj -c Release -o /app
FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "IgniteGarbledDataReproducer.dll"]
docker-compose.yml
version: '3.4'
services:
ignitegarbleddatareproducer:
image: ${DOCKER_REGISTRY-}ignite-garbled-data-reproducer
build:
context: .
dockerfile: IgniteGarbledDataReproducer/Dockerfile
docker-compose-override.yml
version: '3.4'
services:
ignitegarbleddatareproducer:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- IGNITE_CONNECTION_STRING=DRIVER={Apache Ignite};ADDRESS=my-ignite-3e8b8cb9a4e192af.elb.us-east-1.amazonaws.com:10800;SSL_MODE=disable;
Build
docker build . -t ignitegarbleddatareproducer -f IgniteGarbledDataReproducer/Dockerfile
Execute
PS C:\IgniteGarbledDataReproducer> docker-compose up
Starting ignitegarbleddatareproducer_ignitegarbleddatareproducer_1 ... done
Attaching to ignitegarbleddatareproducer_ignitegarbleddatareproducer_1
ignitegarbleddatareproducer_1 | START
ignitegarbleddatareproducer_1 | DRIVER={Apache Ignite};ADDRESS=my-ignite-3e8b8cb9a4e192af.elb.us-east-1.amazonaws.com:10800;SSL_MODE=disable;
ignitegarbleddatareproducer_1 | Connection Opened
ignitegarbleddatareproducer_1 | DROP TABLE IF EXISTS MyTable;
ignitegarbleddatareproducer_1 | Table Dropped (if it existed)
ignitegarbleddatareproducer_1 | CREATE TABLE MyTable (Id varchar(36) not null, FirstName varchar(255), LastName varchar(255), PRIMARY KEY (Id));
ignitegarbleddatareproducer_1 | Table Created
ignitegarbleddatareproducer_1 | INSERT INTO MyTable (Id) VALUES ('80579d98-9010-4610-b12e-ed33ed7d3c62');
ignitegarbleddatareproducer_1 | Data Inserted
ignitegarbleddatareproducer_1 | SELECT Id FROM MyTable;
ignitegarbleddatareproducer_1 | Id = '8 0 5 7 9 d 9 8 - '
ignitegarbleddatareproducer_1 | Data Returned
ignitegarbleddatareproducer_1 | DONE
ignitegarbleddatareproducer_ignitegarbleddatareproducer_1 exited with code 0
UPDATE 2
I turned on ODBC logging by adding the following to /src/apache-ignite-2.7.0-bin/platforms/cpp/odbc/install/ignite-odbc-install.ini and you can get the logs here.
[ODBC]
Trace=yes
TraceFile=/logs/odbc.log
The end of the log after the SELECT Id From MyTable; statement is below for convenience.
[ODBC][1][1556564832.827181][SQLExecDirectW.c][177]
Entry:
Statement = 0xf27a80
SQL = [SELECT Id FROM MyTable;][length = 23 (SQL_NTS)]
[ODBC][1][1556564832.847882][SQLExecDirectW.c][445]
Exit:[SQL_SUCCESS]
[ODBC][1][1556564832.852889][SQLRowCount.c][173]
Entry:
Statement = 0xf27a80
Row Count = 0x7ffd7da98f00
[ODBC][1][1556564832.856831][SQLRowCount.c][247]
Exit:[SQL_SUCCESS]
Row Count = 0x7ffd7da98f00 -> 0
[ODBC][1][1556564832.860335][SQLNumResultCols.c][156]
Entry:
Statement = 0xf27a80
Column Count = 0x7ffd7da98f50
[ODBC][1][1556564832.864944][SQLNumResultCols.c][251]
Exit:[SQL_SUCCESS]
Count = 0x7ffd7da98f50 -> 1
[ODBC][1][1556564832.869909][SQLFetch.c][162]
Entry:
Statement = 0xf27a80
[ODBC][1][1556564832.884876][SQLFetch.c][351]
Exit:[SQL_SUCCESS]
[ODBC][1][1556564832.894173][SQLColAttributeW.c][156]
Entry:
Statement = 0xf27a80
Column Number = 1
Field Identifier = SQL_DESC_NAME
Character Attr = 0xf2e190
Buffer Length = 4096
String Length = 0x7ffd7da98f20
Numeric Attribute = 0x7ffd7da98ee0
[ODBC][1][1556564832.898409][SQLColAttributeW.c][523]
Exit:[SQL_SUCCESS]
[ODBC][1][1556564832.910800][SQLColAttributeW.c][156]
Entry:
Statement = 0xf27a80
Column Number = 1
Field Identifier = SQL_DESC_CONCISE_TYPE
Character Attr = 0xf2e190
Buffer Length = 4096
String Length = 0x7ffd7da98f40
Numeric Attribute = 0x7ffd7da98f00
[ODBC][1][1556564832.914551][SQLColAttributeW.c][523]
Exit:[SQL_SUCCESS]
[ODBC][1][1556564832.925913][SQLGetData.c][237]
Entry:
Statement = 0xf27a80
Column Number = 1
Target Type = -8 SQL_WCHAR
Buffer Length = 4094
Target Value = 0xf2e190
StrLen Or Ind = 0x7ffd7da98ee0
[ODBC][1][1556564832.929959][SQLGetData.c][534]
Exit:[SQL_SUCCESS]
Buffer = [8](unicode)
Strlen Or Ind = 0x7ffd7da98ee0 -> 36
[ODBC][1][1556564832.934835][SQLFetch.c][162]
Entry:
Statement = 0xf27a80
[ODBC][1][1556564832.938930][SQLFetch.c][351]
Exit:[SQL_NO_DATA]
[ODBC][1][1556564832.943970][SQLMoreResults.c][162]
Entry:
Statement = 0xf27a80
[ODBC][1][1556564832.960311][SQLMoreResults.c][345]
Exit:[SQL_NO_DATA]
[ODBC][1][1556564832.965960][SQLFreeStmt.c][144]
Entry:
Statement = 0xf27a80
Option = 0
[ODBC][1][1556564832.970025][SQLFreeStmt.c][266]
Exit:[SQL_SUCCESS]
[ODBC][1][1556564832.975135][SQLFreeHandle.c][381]
Entry:
Handle Type = 3
Input Handle = 0xf27a80
[ODBC][1][1556564832.981240][SQLFreeHandle.c][494]
Exit:[SQL_SUCCESS]
[ODBC][1][1556564832.991337][SQLDisconnect.c][208]
Entry:
Connection = 0xf0a6f0
[ODBC][1][1556564832.996778][SQLDisconnect.c][379]
Exit:[SQL_SUCCESS]
[ODBC][1][1556564833.001505][SQLFreeHandle.c][284]
Entry:
Handle Type = 2
Input Handle = 0xf0a6f0
[ODBC][1][1556564833.006038][SQLFreeHandle.c][333]
Exit:[SQL_SUCCESS]
UPDATE 3
I have a hunch that it might have something to do with UTF-16 due to a similar question where characters are separated with a \0.
Maybe something to do with the UNICODE setting in this line from the ODBC logs?
[ODBC][1][1556564832.214691][SQLDriverConnectW.c][290]
Entry:
Connection = 0xf0a6f0
Window Hdl = (nil)
Str In = [DRIVER={Apache Ignite};ADDRESS=my-ignite-3e8b8cb9a4e192af.elb.us-east-1.amazonaws.com:10800;SSL_MODE=disable;][length = 115 (SQL_NTS)]
Str Out = (nil)
Str Out Max = 0
Str Out Ptr = 0x7ffd7da98dd0
Completion = 0
UNICODE Using encoding ASCII 'ANSI_X3.4-1968' and UNICODE 'UCS-2LE'
UPDATE 4
I found this document from Microsoft which said
If the ASCII character encoding is not UTF-8, for example:
UNICODE Using encoding ASCII 'ISO8859-1' and UNICODE 'UCS-2LE'There is more than one Driver Manager installed and your application is using the wrong one, or the Driver Manager was not built correctly.
I'm assuming my driver isn't built correctly.
And this document said to use --enable-iconv --with-iconv-char-enc=UTF8 --with-iconv-ucode-enc=UTF16LE.
UPDATE 5
I also found this git issue related to .Net Core, unixODBC, and UTF16.
UPDATE 6
I tried adding --enable-iconv --with-iconv-char-enc=UTF8 --with-iconv-ucode-enc=UTF16LE when calling unixODBC's configure, which changed the ODBC connection to UNICODE Using encoding ASCII 'UTF8' and UNICODE 'UTF16LE', but I got the same bad data back regardless. I also tried --with-iconv-ucode-enc=UNICODE, but no change.
This is how I changed my Dockerfile. I removed unixodbc from apt-get and added the following to my Dockerfile
ADD ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.7.tar.gz .
RUN gunzip unixODBC*.tar.gz && tar xvf unixODBC*.tar
WORKDIR /src/unixODBC-2.3.7
RUN ./configure --enable-gui=no --enable-drivers=no --enable-iconv --with-iconv-char-enc=UTF8 --with-iconv-ucode-enc=UTF16LE
RUN make && make install
ENV LD_LIBRARY_PATH="/usr/local/lib"
And this is what the odbc log looks like now:
[ODBC][7][1556723781.839804][SQLDriverConnectW.c][290]
Entry:
Connection = 0x1ddff70
Window Hdl = (nil)
Str In = [DRIVER={Apache Ignite};ADDRESS=my-ignite-3e8b8cb9a4e192af.elb.us-east-1.amazonaws.com:10800;SSL_MODE=disable;][length = 115 (SQL_NTS)]
Str Out = (nil)
Str Out Max = 0
Str Out Ptr = 0x7ffeca3331a0
Completion = 0
UNICODE Using encoding ASCII 'UTF8' and UNICODE 'UTF16LE'
UPDATE 7
I connected via isql and got the correct response, but when I connected via iusql I got the "bad data" from my reproducer.
UPDATE 8
Adding some odbc installation info:
root@4efdb5ed98b5:/app# odbc_config --cflags
-DHAVE_UNISTD_H -DHAVE_PWD_H -DHAVE_SYS_TYPES_H -DHAVE_LONG_LONG -DSIZEOF_LONG_INT=8 -I/usr/local/include
root@4efdb5ed98b5:/app# odbc_config --ulen
-DSIZEOF_SQLULEN=8
root@4efdb5ed98b5:/app# odbcinst -j
unixODBC 2.3.7
DRIVERS............: /usr/local/etc/odbcinst.ini
SYSTEM DATA SOURCES: /usr/local/etc/odbc.ini
FILE DATA SOURCES..: /usr/local/etc/ODBCDataSources
USER DATA SOURCES..: /root/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
Solution
This is a bug with Apache Ignite 2.7.0, so there is no answer. An issue has been filed with Apache Ignite to fix it.
- How to register multiple implementations of the same interface with different dependencies
- How to do async Windows Service ExecuteAsync properly?
- Entity Framework Core 3.1.6 Item With Same Key Has Already Been Added
- How to pass type parameter to Body() when render fragment?
- Bind gRPC services to specific port in aspnetcore
- Rollback() but the query still getting new value
- .NET Core Dependency Injection with inherited interface
- Launch Visual Studio Code Programmatically
- How to get results from a Postgresql function using EF Core?
- Why string property must be declared as nullable string in Entity Framework Core when corresponding database table column is nullable
- What is the default location for certificates created using "dotnet dev-certs https"
- Runing a functions project locally
- Why do I get "The response ended prematurely" when using the HttpClient?
- Using the Google Workspace API to get the storage used by a user
- How to force Visual Studio to re-create the SSL certificate for a .NET Core Web Application running Kestrel?
- Using System.Text.Json to Serialize an IConfiguration back to Json
- How to mock service injected by HttpContext.RequestServices.GetService<> in derived controller?
- .NET Core Error 1053 the service did not respond to the start or control request in a timely fashion
- Using app.config in .Net Core
- How to keep a schedule alive (created using Quartz & .Net Core) - as it is not working after some time after deployed on IIS
- Seeking an explanation for the change in order of widening operations from .NET Framework 4.8 to .NET 8
- Entity Framework Core automatically inserts double quotes in Oracle database
- VS Code / OmniSharp failing to load project; Can't find a package that is present
- Is it good to create Startup.cs in .NET Core Projects rather than registering services in Program.cs?
- Trouble with ActionResult in unit test can't Assert.Equal on POST return value (asp.net core mvc web api)
- Is .NET Core 3 (and .NET 5 to .NET 7) compatible with Windows 7?
- Mudblazor Select with multiselect and Fluentvalidation For-Expression
- System.Text.Json: How do I specify a custom name for an enum value?
- .NET Core MVC Page Not Refreshing After Changes
- Test for schema changes in Entity Framework migration