libmemcached : SERVER HAS FAILED AND DISABLED UNTIL TIMED RETRY

Question

I am using libmemcached C API for interacting with memcached. Occasionally memcached_set fails with SERVER HAS FAILED AND DISABLED UNTIL TIMED RETRY. What can be a general approach to handle this issue.

This error only occurs on production servers, and it is not producible in development environment.

• memcached version : 1.4.7
• libmemcahced version: 1.0.2
• Linux RHEL6 OS
• GCC 4.7.3

Following is the pseudo code snippet, cannot post entire code.

bool set_keys(std::string const & query)
{
  // create a DB connection
  // execute query
  // consider result of query is as following
  std::vector<std::tuple<std::string,std::string>>result_set;
  memcached_return rc;
  for(auto const & item :  result_set )
  {
      rc= memcached_set(memc, std::get<0>(item).c_str(), std::get<0>(item).size(),  std::get<1>(item).c_str(),  std::get<1>(item).size(), (time_t)0, (uint32_t)0);

       if (rc != MEMCACHED_SUCCESS)
            fprintf(stderr,"Couldn't store key: %s\n",memcached_strerror(memc, rc));
   }
}

One thing I noticed that memcached_server_list_append is not used anywhere in code after creations of memcached connection, can this cause issues?

Answer 1

Possible the server is overloaded. Check the loading of memcached by top.

Check the key names when set for valid values.

Try to set Memcached::OPT_BINARY_PROTOCOL, to enable binary data.