I've been looking at performance tuning an environment and I've setup a number of parameters to improve NFS performance, from the simple things like setting mount options, to the advanced like setting sysctl parameters and even considered enabling jumbo frames on the NICs and firewall.
I'm running NFS and memcached on the same server, and I feel like I could possibly gain a performance improvement by removing the memcached networking overhead.
Some environment info is in order first:
4 load balanced web servers in a dsr setup, connected to an ASA 5510. 2 mysql servers in master/master replication. 1 app/file server running memcached and nfs.
Adding additional servers is not on the table for discussion.
Web servers serve their content off of exported NFS mounts. NFS mounts on the web servers are setup to use udp instead of tcp.
Web servers run varnish in front of apache/php, varnish is setup to only serve images, css, and javascript, via separate subdomains for each.
Site loads within 5 seconds most of the time, but occasionally, even at night when traffic is low, we will see it take longer to load. Sometimes its up to 10 seconds. Sure that's not bad, but there's no reason we shouldn't be able to see better performance. All files are served over NFS. By the way I acquired this customer recently, after their original tech switched positions.
What I thought I could/should do since the server is optimized now for NFS connections, was to setup memcached to run in unix socket mode, which disables its network functionality, and then place the unix socket on an exported filesystem. That way the web servers could still connect to memcached, but it would remove some of the overhead because I have NFS mounts running udp instead of tcp.
A unix socket file on the filesystem is merely a handle for two processes to connect to the same socket; all of the communication is local, and putting it on NFS (if that's even possible) won't enable communication across machines. Besides, even if NFS had some mechanism, it would likely be worse than using memcache directly.