Search code examples
redisperformance-testingredis-clustergo-redis

Redis taking too much time to execute simple commands during concurrent access

Background of problem

Hello all , i have made a project in golang gin , and I have integrated the redis Clusterclient in it using "github.com/go-redis/redis/v7"

P.S. Redis that I am using is a redis cluster hosted on AWS

the redis commands that I am using are simply redis.Get and redis.Set only.

Now I have made one API and used caching in it, and when I run it locally, response times are around 200 to 300ms, which is awesome (thanks to Redis)

Main Problem

now when I start doing the load testing on the same API with around 100 concurrent users , response time gets significantly increased ( around 4 seconds). I used spans to monitor the time taken by a different part of the code, and I got this

enter image description here

Getting from primary, getting from secondary are for the redis.Get command

Setting the primary , setting the secondary are for redis.Set

both commands are taking around 1 sec to execute, which is unacceptable,

can anyone please tell me some way, so that I can tackle this problem and reduce the time for the redis commands to execute

Solution

  • Ok so I have solved this somehow. Firstly I have updated my golang redis client library from go-redis/v7 to go-redis/v8 . And it made a significant improvement. I will advise everyone to do the same.

    But still I am suffering from high response time , so the next step for me wa sto change the redis infra. Earlier I was using a redis cluster which only had 1 shard, but now I have moved to another redis having 4 shards..

    And it made a huge difference , my response goes from 1200ms to 180ms. Kindly note that these response time are coming when I am doing a load testing with 100 concurrent users with an average of about 130rps

    So in short upgrade your redis client , upgrade your redis infra