i was experimenting with Go worker pools to speed up my import script with concurrent workers. But without any luck by now. And i spend lot of hours until i wrote this post...
First i used this one https://github.com/gammazero/workerpool, which worked out very well in first place. But the problem here, is that i submit Jobs to some 'pool' and these jobs are executed. When i try write to the same SQL Table at the same time from all jobs, i get some Segmentation fault after random amount of time. Then i got the advice to open a new sql connection per worker. And that is by design not possible with the upper library, as i have no control about the workers themselves.
Then i tried the code from here https://gobyexample.com/worker-pools and entered the SQL Open and Close at the top and the bottom of the worker() for-range. But i had no luck. Although i pushed 1000s of jobs to the Pool, only 16 jobs where taken (1 by each worker, with 16 workers) and stop. And i have no clue why.
Could anyone of you provide me a solid example code about a golang worker pool, where each worker controls the SQL Connection ?
By the way, i use https://gorm.io as MySQL ORM, which works out very well up to this point.
Maybe anyone of you could help me. I can also provide you with further details. That would speed up my import script massively.
Bye then Adrian
i finally found out, that the opening and closing of the mysql connection didnt cause the segfault. it was rather the data inside some of the products. after adding a defer func to recover from segfaults everyting is running smoothly.