Azure VM scale set diagnostic ETW creating duplicate entries due to being unable to read log timestamps

Question

I have Service Fabric cluster which is underpinned by a 5 node VM scale set. I have enabled the Diagnostic extension on the scale set and have configured it to enable transfer of ETW logs and send the data to Application Insights.

I can see that this is happening but I am seeing duplicate entries for each message generated. Every message is shown in App Insights 4 times:

I am also logging to table storage via a listener in the code and can see that the traces are only recorded once there, so I am pretty sure it is an issue with the diagnostic extension.

I have noticed the following warnings in the ETW log however:

It seems to suggest that the duplicates are being caused by an inability to read the log timestamp as it is being accessed by something else at the same time.

Is this because there are multiple nodes on the scale set and they are all trying to access the ETWEventTable at the same time to get the timestamp?

Is there a way to fix this? Or if another issue is the cause how can I get around it?

Answer 1

The problem here was that I had multiple providers all using the same event destination:

    "EtwEventSourceProviderConfiguration":[  
          {  
              "provider":"Provider1",
              "Event":[  

              ],
              "DefaultEvents":{  
                 "eventDestination":"ETWEventTable"
              }
           },
           {  
              "provider":"Provider2",
              "Event":[  

              ],
              "DefaultEvents":{  
                 "eventDestination":"ETWEventTable"
              }
           },
           {  
              "provider":"Provider3",
              "Event":[  

              ],
              "DefaultEvents":{  
                 "eventDestination":"ETWEventTable"
              }
           }
        ],

I added uniquely named event destinations and that seems to have stopped the messages duplicating:

 "EtwEventSourceProviderConfiguration":[  
          {  
              "provider":"Provider1",
              "Event":[  

              ],
              "DefaultEvents":{  
                 "eventDestination":"ETWEventTableProvider1"
              }
           },
           {  
              "provider":"Provider2",
              "Event":[  

              ],
              "DefaultEvents":{  
                 "eventDestination":"ETWEventTableProvider2"
              }
           },
           {  
              "provider":"Provider3",
              "Event":[  

              ],
              "DefaultEvents":{  
                 "eventDestination":"ETWEventTableProvider3"
              }
           }
        ],