How deal with big trace which can be divise in OpenTelemetry?

Question

I have a worker, which call a function foo every x secs. I would like to trace every call of this function foo, to monitoring how things go to each call.

I am using c# activity to monitoring, and it goes in OpenTelemetry collector / backend / etc... I watch these trace on Grafana.

My function foo is similar to :

void foo()
{
  using var activity = activitySource.StartActivity("foo");

  var userIDList = GetUserIDList(); // Some span are create here

  foreach (var userID in userIDList)
  {
    someActionOnUser(userID); // lot of span are create here
  }
}

The final trace is heavy (something like 10Mo), and I would like to reduce this size because backend has a size limit (I could grow up this limit, but I search a better solution...).

My idea was to :

• Create a parent trace A for each loop iteration (which will catch span create in someActionOnUser)
• Create a span B for parent trace "foo" (which will not catch span create in someActionOnUser)
• Set up a Span link from A to B and B to A

In C#, Span link is Activity Link.

But in C#, you can set Activity Link only when you create an activity (link to activity code source), so this solution is not really possible...

I found this discussion, with the suggestion to add a function AddLinks in activity, and the problem with this.

So I suppose this idea is not really good (I wonder if it's not an anti-pattern...). I don't think I'm the first one with this problem, but I don't find many resources about this. I suspect I have a bad strategy to deal with that.

How people deal with this problem ?

Answer 1

https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/links-creation-with-new-activities - Check this doc, it has some examples on how to achieve this. (Note: AddLink API is coming later this year : https://github.com/open-telemetry/opentelemetry-dotnet/issues/5273)