Search code examples
network-programmingcpu-architecturemulticoremotherboard

How do the CPUs on different sockets communicate?

I am tuning the performance of my parallel Java program. I am curious about the Architecture effects.

Given a machine with two CPU sockets, each one with a quad-core Intel Xeon CPU, then:

  • How do the two CPUs communicate, how fast would they communicate?
  • How fast would two cores on the same chip communicate?
  • Are the four cores on the same chip equivalent in terms of communicating or memory accessing?
Solution

  • 1) How do the two CPUs communicate, how fast would they communicate?

    Most time they communicate via memory or nearest shared memory hierarchy level. (System memory both on SMP and NUMA is considered as shared level; even if in NUMA it is accesses via memory controller of another chip. this is just Non-Uniform=slower access)

    2) How fast would two cores on the same chip communicate?

    Cores on same chip usually shares L2 or L3 cache. Cores on different chips communicate via memory or with cache-to-cache interactions using cache coherency protocol.

    So in case 1 (different chips) speed (bandwidth) of memory passing between CPUs will be near plain memory read/write. And in case 2 (same chip) this speed can be bigger, up to cache read/write speed.

    Latency of communication will be several hundreds of CPU ticks in case 1 and several dozens in case 2.

    3) Are the four cores on the same chip equivalent in terms of communicating or memory accessing?

    All four cores of same chip usually have equivalent distance to RAM. It depends on chip architecture and implementation; for some older Intels e.g. multicore chip was really two chips packed into single package.