Reduce access time to Bigtable with Golang

Question

I would like to reduce the access time to my table on my Golang RestApi.

I have a Go restAPI with an enpoint that requires access to a bigtable database for each code specified at the request body. My design of the key access is "provider|client|productCode|date|.." (and other 4 parameters).

For each productCode of the request body I'm making a async call to read the row with a specified key.
On my test I requested 1000 products codes and the obtained times on my local machine are around of 800~900ms. The times on a cloud machine are around 550~450 ms.

I'm using the bigtable package to access to the bigTable and I'm using the following code:

package main

import (
    "log"
    "time"

    "golang.org/x/net/context"
    "google.golang.org/cloud/bigtable"
)

func main() {
    start2 := time.Now()

    listPKs := []string{"PROV|CLI|6030|20160714|8|30301.30302|ES", "PROV|CLI|44103|20160714|8|30301.30302|ES", "PROV|CLI|1454871|20160714|8|30301.30302|ES"}

    providerRS := getBDresponse(listPKs, 50000)
    if providerRS != nil {
        ///do something
    }
    elapsed2 := time.Since(start2)
    log.Printf("Time access BT: %s", elapsed2)
}

func getBDresponse(listPKs []string, timeoutMiliseconds int) []string {

    resp := make(chan string)
    table := myClient.Client.Open(TABLE_NAME)
    //async call to BT
    for _, key := range listPKs {
        go asyncDoUniqueCall(key, resp, myContext, table)
    }
    //get all responses
    providerRS := getResponses(resp, timeoutMiliseconds, len(listPKs))
    return providerRS
}

func asyncDoUniqueCall(pk string, ch chan string, ctx *context.Context, table *bigtable.Table) {
    ch <- GetRowValue(pk, ctx, table)
}

func getResponses(resp chan string, timeoutMiliseconds int, totalNumRQ int) []string {
    var ret []string

    for j := 0; j < totalNumRQ; j++ {
        select {
        case rowResponse := <-resp: //catch the response
            if rowResponse != "" {
                ret = append(ret, rowResponse)
            }
        case <-time.After(time.Duration(timeoutMiliseconds) * time.Millisecond): // timeout control 
            return nil
        }
    }
    return ret
}

//GetRowValue open the table and get the row indexed by pk and returns the stored message 
func GetRowValue(pk string, ctx *context.Context, table *bigtable.Table) string {

    var response string

    r, err := table.ReadRow(*ctx, pk)
    if err != nil {
        return ""
    }
    if len(r) > 0 {
        row := r[COLUMN_FAMILY_NAME]
        numCol := len(row)

        response = string(row[0].Value)
    }
    return response
}

I tried this example too and the times are not better:

Am I using the gorutines and chanels correctyl? Is the way how I'm accessing to the BT correct? Maybe the key design is not correct?

Answer 1

I think the problem is that you are using a 1 dimension chan:

resp := make(chan string) 

You should try to enlarge its dimension by doing:

resp := make(chan string,len(listPKs))

That's because if your async function finishes before the info in the chan is read than the chan will be blocking.

A side note: spawning go routines without performing any check could be a possible source of problems.

I'd suggest you to implement a working queue in order to optimise your task.