I'm writing a server in erlang that handles very large amount of messages (records). Each message has a tag (atom) like a user id.
The "router" will spawn a dedicated-permanent process for this user (to accumulate messages for few minutes before saving them and passing them) if such process doesn't exists. Otherwise it will pass it to the existing process mailbox as message.
The problem is the bookkeeping of a routing table.
I could think of serializing the router, each message will result in ETS lookup to find a PID by userId and eventually Spawn and ETS insert if it didn't exit. But that got congested in few seconds.
An alternative is to spawn a process directly to route each message, but that might result in race condition if a few messages to a single user came in in close sucession and didn't find their corresponding PID in ETS and spawned permanent processes. Messages will be lost and only the last spawned process will be valid (overwrite others in ETS) where the others will go idle and untracked.
I also might be thinking all wrong. is there a better way to handle this scenario?
It is a frequent pattern to have one process (one server) per user, just like you propose.
Sometimes, if the protocol used allows it, instead of routing the messages to a "user" process from a server listening all the users, a new process is spawned before any connection, listening (waiting) for a new user connection request. When a request arrives, a new waiting process is spawned, and the current one manages the complete session with the new user (see learyousomeerlang: a bucket of socket for a detailed example).