Search code examples
debuggingerlangtracebeam

Who is the sender of Erlang's trace messages and what can I assume based on it?


When using erlang:trace/3, the tracing process receives "trace messages" which tell the story of the traced process. For the purpose of message ordering (i.e. the guarantee that messages between fixed sender and recipient are delivered in order, if delivered), who is the sender of trace messages?

For instance, if a traced process sends a message to p1 and then to p2, do I have a guarantee that the tracing process will receive the notification about p1 before p2?

For another instance, if the traced process looks as follows

proc() ->
  p1 ! ok,
  receive X -> ok end,
  p2 ! ok. 

Does the 'receive' trace message always come exactly between send to p1 and p2?

Yet another doubt I have is what if there are more processes traced by the same tracer:

proc1() ->
  proc2 ! ok.

proc2() ->
  receive ok -> ok end.

Here proc1 is guaranteed to send before proc2 receives. Can I infer that if I trace both proc1 and proc2, I will be informed about those events in this exact order?


The post may look like many questions, but the common denominator is that I would like to know what I can assume about the relationship between the order of events in the system and the order of observations by the tracer. I could not find any mentions of that matter in the documentation, but I might have missed it somehow.


Solution

  • The only ordering guarantees are for a single process. You can view it as for every traced process, there is another process sending trace messages to the trace receiver. Depending on lock conflicts and other scheduling details in between processes, trace messages may arrive in unpredictable orders. All trace messages except receive from the same process are however predictable.

    As for your examples:

    For instance, if a traced process sends a message to p1 and then to p2, do I have a guarantee that the tracing process will receive the notification about p1 before p2?

    The send trace messages from the sender are guaranteed to arrive in order, but the receive trace messages from the receivers are not. Any other trace messages generated by the sender are also guaranteed to be in order, the only odd one being receive as explained below.

    For another instance, if the traced process looks as follows

    proc() ->
      p1 ! ok,
      receive X -> ok end,
      p2 ! ok.
    

    Does the 'receive' trace message always come exactly between send to p1 and p2?

    The receive trace happens when a message is placed in the mailbox, not when it is removed from it. So it is guaranteed to be before p2 ! ok, but could be before or after p1 ! ok.

    Yet another doubt I have is what if there are more processes traced by the > same tracer:

    proc1() ->
      proc2 ! ok.
    
    proc2() ->
      receive ok -> ok end.
    

    Here proc1 is guaranteed to send before proc2 receives. Can I infer that if I trace both proc1 and proc2, I will be informed about those events in this exact order?

    No. The receive trace message can arrive before the send. It is unlikely in the current implementation, but it could happen due to different race conditions inside the VM.

    I could not find any mentions of that matter in the documentation, but I might have missed it somehow.

    I thought I had documented this better when I re-implemented the tracer implementation a couple of years ago, but seems like even I cannot find it so probably it does not exist. Maybe I should take some time to write all this down...