How to group logs by "start-point" among a distributed system

Question

I have a distributed system with lots of machines, each machine produces logs, and can call services in others machines (that produces logs too).

I'm using a centralized log service (Logentries), and what I have is this:

12:00:00 Server1 apache log
12:00:01 Server1 application log
12:00:01 Server1 apache log
12:00:02 Server2 Some service log
12:00:02 Server1 application log
12:00:03 Server2 Some service log

but what I really want is this:

 12:00:00 Server1 apache
 12:00:01 Server1 application log
 12:00:02 Server2 Some service log

 12:00:01 Server1 apache
 12:00:02 Server1 application log
 12:00:03 Server2 Some service log

These logs are grouped by the start point (the apache log).

There are any solution to do that? I can stop use logentries and use other Log Management SaaS.

Answer 1

You don’t have this information in the logs, so you can’t group by it. You could generate an ID, probably a GUID and log it together with every other message. This way you’d know the execution path.

I’m not sure how your logs are being sent to the centralized system, but if asynchronously, you’d also need to provide it with a logical clock (lamport clock) if you jump between different instances and services, because the order in which they’d arrive at the central server can change.