Search code examples
sql-serverbcp

"Copy in failure" issue with BCP into SQL Server (seems unrelated to data overflow)?


Trying to load data from TSV in linux machine into a SQL Server database using bcp from mssql-tools package on CentOS 7. Getting an error:

BCP copy in failure

and no errors produced in the bcp-generated error files (when using the -e option). This makes me think it is a problem with this data set on the SQL Server database side.

Note, I am using similar BCP command to write data to the same database for other TSV data sets.

An example of the trace logs that I see for the process in SQL Server Mgnt Studio is shown below (not sure what to make of these logs, but hopefully are relevant)

Audit Login -- network protocol: TCP/IP
set quoted_identifier on
set arithabort off
set numeric_roundabort off
set ansi_warnings on
set ansi_padding on
set ansi_nulls on
set concat_null_yields_null on
set cursor_close_on_commit off
set implicit_transactions off
set language us_english
set dateformat mdy
set datefirst 7
set transaction isolation level read committed
myuser


SQL:BatchStarting    set quoted_identifier off          myuser          

SQL:BatchCompleted   set quoted_identifier off          myuser

RPC:Completed   exec sp_describe_first_result_set N'select * from NOTE_ENC_INFO'            myuser

RPC:Completed   declare @p1 int
set @p1=1
exec sp_prepare @p1 output,NULL,N'select * from NOTE_ENC_INFO',1
select @p1          myuser

RPC:Completed   exec sp_describe_first_result_set N'select * from NOTE_ENC_INFO'            myuser  

SQL:BatchStarting   insert bulk NOTE_ENC_INFO([NOTE_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[CONTACT_DATE_REAL] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_INSTANT_DTTM] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGNUSER_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_NOTE_LINK] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_REQUIRED_C] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[etl_date] datetime)           myuser

SQL:BatchCompleted  insert bulk NOTE_ENC_INFO([NOTE_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[CONTACT_DATE_REAL] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_INSTANT_DTTM] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGNUSER_ID] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_NOTE_LINK] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[COSIGN_REQUIRED_C] varchar(254) collate SQL_Latin1_General_CP1_CI_AS,[etl_date] datetime)           myuser      

RPC:Completed   exec sp_unprepare 1         myuser  

Note that in the trace it logs that the SQL:BatchStarting events generate 2 reads, but 0 writes (whereas I would think they should be generating > 0 writes for writing the actual TSV data to the DB table (not very experienced with this stuff though, so really IDK)).

For reference, the bcp command I'm running looks like:

TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
RECOMMEDED_IMPORT_MODE='-c' # makes a big difference, see https://stackoverflow.com/a/16310219/8236733
/opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
        $TO_SERVER_ODBCDSN \
        -U $USER -P $PASSWORD \
        -d $DB \
        $RECOMMEDED_IMPORT_MODE \
        -t "\t" \
        -e ${filename}.bcperror.log

(Note "-D option: Causes the value passed to the bcp -S option to be interpreted as a data source name (DSN)." I use this for using a /etc/odbc.ini file rather than a IP string literal. See https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/connecting-with-bcp?view=sql-server-2017)


Solution

  • Found the reason: The SQL Server table design was set such that not fields could be null and apparently was causing bcp to just quit when attempting to insert a record that did have nulls, once I set the table design so that the appropriate fields could be null the bcp process completed as expected.

    * Note sure how I would have detected this from the bcp output or the SQL trace, so if anyone has any better debugging tips or how I could have caught this error earlier please do let me know.