Memcached / Dalli couldn't fetch data from other servers

Question

We use memcached as session store for our application, we have 2 production servers and the memcached client is dalli (https://github.com/mperham/dalli). The config is following:

# Server A
config.cache_store = :dalli_store, '127.0.0.1', SERVER_B_IP, {namespace: 'Myapp', expires_in: 1.day, compress: true, failover: false}
# Server B
config.cache_store = :dalli_store, SERVER_A_IP, '127.0.0.1', {namespace: 'Myapp', expires_in: 1.day, compress: true, failover: false}

We're facing a problem that one of our servers seems not to look up the other one to fetching data. For example, if I have data for some_key on server B, when I tried to debug in console:

options = {namespace: 'Myapp', expires_in: 1.day, compress: true, failover: false}
# Server A
dalli = Dalli::Client.new(['127.0.0.1', SERVER_B_IP], options)
dalli.get("some_key") # nil

# Server B
dalli = Dalli::Client.new([SERVER_A_IP, '127.0.0.1'], options)
dalli.get("some_key") # { "_csrf_token" => "..." }

I tried with failover: true but still got the same result. But if the key some_key stay on server A, I can get the same data from both servers.

Also, i the above example, if I initialize dalli client with only SERVER_B_IP, i.e dalli = Dalli::Client.new(SERVER_B_IP, options), I could get back the data.

Digging the code a little bit, I found out the ring.continuum on server A always point to itself:

# Server B
ring = dalli.send :ring
hkey = ring.send :hash_for, 'some_key'                    # got same hkey
entryidx = ring.send :binary_search, ring.continuum, hkey # 300
ring.continuum[entryidx].server                           # server B
ring.continuum[302].server                                # server A

# Server A
ring = dalli.send :ring
hkey = ring.send :hash_for, 'some_key'                    # got same hkey
entryidx = ring.send :binary_search, ring.continuum, hkey # 302
ring.continuum[entryidx].server                           # server A
ring.continuum[300].server                                # server A

Am I missed some thing in the memcached / dalli config?

Thanks in advance.

Answer 1

I had quick look at Daili code base. It uses consistent hashing for distributing keys to servers.

https://github.com/mperham/dalli/blob/master/lib%2Fdalli%2Fring.rb#L10

entry_count_for(server, servers.size, total_weight).times do |idx|
     hash = Digest::SHA1.hexdigest("#{server.name}:#{idx}")
     value = Integer("0x#{hash[0..7]}")
     continuum << Dalli::Ring::Entry.new(value, server)
end

and

def entry_count_for(server, total_servers, total_weight)
     ((total_servers * POINTS_PER_SERVER * server.weight) / Float(total_weight)).floor
end

The keyspace of each server depends on total servers, weight and server name. So that the keyspace will be different with different total servers or server name. I think that explanation fits your problem.