I'm inserting simple dataframe with Glue to RDS Sql Server instance.
Last row (of 8 in total) doesn't get inserted and throws error:
FAILED: table_name, Exception: An error occurred while calling o216.pyWriteDynamicFrame. : org.apache.spark.SparkException:
Job aborted due to stage failure: Task 15 in stage 4.0 failed 4 times, most recent failure: Lost task 15.3 in stage 4.0 (TID 35, 10.234.42.138, executor 1): java.sql.BatchUpdateException: A result set was generated for update.
I checked the structure of the table, constraints, indexes, triggers between failing environment (dev) and remaining ones and did not find any differences. Also I'm excluding issues with Glue/Python as my function.py in the job is used in all environments.
From SQ: Server profiler I can confirm that insert is being retried 4 times and got additional logs:
declare @p1 int
set @p1=1
exec sp_prepexec @p1 output,N'@P0 nvarchar(4000),@P1 nvarchar(4000),@P2 bigint,N'INSERT INTO [schema].[table] ("col","col","col") VALUES (@P0,@P1,@P2) ',N'xyz',N'xyz',7,N'xyz'
select @p1
exec sp_unprepare 1
IF @@TRANCOUNT > 0 ROLLBACK TRAN
Thanks for suggestions on where to go further with this...
For the time being below workaround did the job: