fast data exchange between containers hosted in kubernetes

I am working on moving a application which require near real time exchange of data between processes running in multiple containers in kubernetes cluster. I am thinking of using redis cache for this purpose.

The type of data that needs to be exchanges are simple types like double,string values. The frequency of exchange needs to near real time(sub seconds)

are there any other more performant mechanisms available for exchanging data between containers hosted in kubernetes environment?

Solution

This is hugely complex question with way more nuance than can fit in here. It very much depends on the object sizes, uptime requirements, cluster scale, etc. I would recommend you try all of them, evaluate performance, and analyze failure modes as they apply to your use case.

Some things you can try out:

Redis
Memcached
Local files with mmap
Network block device with mmap
NFS with mmap
All three of the above with RocksDB
Postgres
Kafka

On the encodings side evaluate:

JSON (don't use this, just for baseline)
ProtocolBuffers
CapnProto
Msgpack
Maybe BSON?