Trying to load data from TSV in linux machine into a SQL Server database using bcp
from mssql-tools
package on CentOS 7. Getting an error:
BCP copy in failure
and no errors produced in the bcp-generated error files (when using the -e
option). This makes me think it is a problem with this data set on the SQL Server database side.
Note, I am using similar BCP command to write data to the same database for other TSV data sets.
An example of the trace logs that I see for the process in SQL Server Mgnt Studio is shown below (not sure what to make of these logs, but hopefully are relevant)
Audit Login -- network protocol: TCP/IP
set quoted_identifier on
set arithabort off
set numeric_roundabort off
set ansi_warnings on
set ansi_padding on
set ansi_nulls on
set concat_null_yields_null on
set cursor_close_on_commit off
set implicit_transactions off
set language us_english
set dateformat mdy
set datefirst 7
set transaction isolation level read committed
myuser
SQL:BatchStarting set quoted_identifier off myuser
SQL:BatchCompleted set quoted_identifier off myuser
RPC:Completed exec sp_describe_first_result_set N'select * from NOTE_ENC_INFO' myuser
RPC:Completed declare @p1 int
set @p1=1
exec sp_prepare @p1 output,NULL,N'select * from NOTE_ENC_INFO',1
select @p1 myuser
RPC:Completed exec sp_describe_first_result_set N'select * from NOTE_ENC_INFO' myuser
SQL:BatchStarting insert bulk NOTE_ENC_INFO([NOTE_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[CONTACT_DATE_REAL] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_INSTANT_DTTM] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGNUSER_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_NOTE_LINK] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_REQUIRED_C] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[etl_date] datetime) myuser
SQL:BatchCompleted insert bulk NOTE_ENC_INFO([NOTE_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[CONTACT_DATE_REAL] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_INSTANT_DTTM] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGNUSER_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_NOTE_LINK] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_REQUIRED_C] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[etl_date] datetime) myuser
RPC:Completed exec sp_unprepare 1 myuser
Note that in the trace it logs that the SQL:BatchStarting
events generate 2 reads, but 0 writes (whereas I would think they should be generating > 0 writes for writing the actual TSV data to the DB table (not very experienced with this stuff though, so really IDK)).
For reference, the bcp command I'm running looks like:
TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
RECOMMEDED_IMPORT_MODE='-c' # makes a big difference, see https://stackoverflow.com/a/16310219/8236733
/opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
$TO_SERVER_ODBCDSN \
-U $USER -P $PASSWORD \
-d $DB \
$RECOMMEDED_IMPORT_MODE \
-t "\t" \
-e ${filename}.bcperror.log
(Note "-D option: Causes the value passed to the bcp -S option to be interpreted as a data source name (DSN)." I use this for using a /etc/odbc.ini file rather than a IP string literal. See https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/connecting-with-bcp?view=sql-server-2017)
Found the reason: The SQL Server table design was set such that not fields could be null and apparently was causing bcp to just quit when attempting to insert a record that did have nulls, once I set the table design so that the appropriate fields could be null the bcp process completed as expected.
* Note sure how I would have detected this from the bcp output or the SQL trace, so if anyone has any better debugging tips or how I could have caught this error earlier please do let me know.