The MemCached documentation mentions data is distributed across nodes. That's their definition of distributed cached. If a node A needs data that is on node B, data is transferred from B to A. If A crashes, all data stored on A is not available to B anymore.
However, EhCache has a different definition of distributed caching. Basically, it is more like shared-memory than distributed cache. If node A modifies some data, node B will see that modification. If A crashes, any data A stored in the shared-memory remains available to node B.
This leads me to two questions:
If I have 3 nodes A, B, C each having 1GB of memory, it seems like MemCached will add up memory and make it look like a total of 3 GB of memory for the nodes. However, it seems like EhCache does not add the 3 GB, but rather will allow at most 1GB of shared memory between each nodes. Is this correct?
If answer to 1. is yes, then is it correct to conclude that EhCache and MemCached are actually complementary rather than in competition?
I've some experience with both.
I'd say 'yes' to both your questions.
I've been using ehcache together with Hibernate. It caches data found in a central database in memory on several server nodes in a cluster. Reason is of course to have less database accesses. If a cached value is written to by one node, then this value should be invalidated on the other nodes to force them to re-read the value from the database, and re-cache it.
Memcached I've used as datasource. It's a simple key=value database. On every service node you define a list of memcached nodes that it can use. Then it will write to some node (by round robin I guess). The key has the information about which memcached node the value is stored on.
So in my experience, memcached is a, often distributed, database (not the relational kind, but still a database), while ehcache is more a traditional cache.