I am trying to make concurrent HTTP post call to telnyx(message service) API and then update database with the respose. The API has a limit on max call of 200/s for us.Below is the code i came up with after reseaching.It works for small sample test. If somebody can explain if this approach is correct and can deal with large number of calls in production down the line and if i am using the rate,channel and waitgroup correctly in Send()?
func SMSPost(m Message) (*http.Response, error) {
....
return response,nil
}
func Send(sendmessage []Message,db *sql.DB){
rlim := rate.NewLimiter(200, 200)
semaphore := make(chan struct{}, 20)
wg := sync.WaitGroup{}
for i := 0; i < len(sendmessages); i++ {
sendmessage := sendmessages[i]
wg.Add(1)
go func() {
defer wg.Done()
err := rlim.Wait(context.Background())
if err != nil {
log.Println("rate Wait:", err)
}
// Check the concurrency semaphore.
semaphore <- struct{}{}
defer func() { <-semaphore }()
response, err := SMSPost(sendmessage)
/*
...Inserting/updating multiple database table based on response
*/
}()
}
wg.wait()
}
You can use your approach with the semaphorcounter. But try and check before you create the goroutine or else you will create an unlimited number of routines and then wait inside the routine. So just push the 'semaphore <- struct{}{}' up, before you create the goroutine and then you only have a limited number of goroutines at one time:
for i := 0; i < len(sendmessages); i++ {
sendmessage := sendmessages[i]
wg.Add(1)
// Check the concurrency semaphore.
semaphore <- struct{}{}
go func() {
defer wg.Done()
err := rlim.Wait(context.Background())
if err != nil {
log.Println("rate Wait:", err)
}
defer func() { <-semaphore }()
response, err := SMSPost(sendmessage)
/*
...Inserting/updating multiple database table based on response
*/
}()
}