When I run the command show below, I get the logs of the per each transaction made on a single activity (from start to end).
GET /apm-*-transaction/_search
{
"size": 100,
"query": {
"range": {
"@timestamp": {
"gte": "2021-07-19T12:58:40.710-05:00",
"lt": "2021-07-19T13:08:40.710-05:00"
}
}
}
}
Here is a sample (2 out of 5 results) shown below:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "apm-7.13.0-transaction-000002",
"_type" : "_doc",
"_id" : "3CXzv3oB2DAWJifJt8jM",
"_score" : 1.0,
"_source" : {
"agent" : {
"name" : "dotnet",
"version" : "1.11.0+a4eb5008af64ca52fe6973c18810c0c8c7a99554"
},
"source" : {
"ip" : "::1"
},
"processor" : {
"name" : "transaction",
"event" : "transaction"
},
"url" : {
"path" : "/",
"original" : "https://localhost:5001/",
"scheme" : "HTTP",
"domain" : "localhost",
"full" : "https://localhost:5001/"
},
"observer" : {
"hostname" : "ef75b7ae7c69",
"id" : "de63825b-0f59-4df0-aea9-4a538997f6e6",
"type" : "apm-server",
"ephemeral_id" : "d050562a-8d75-4209-b7d0-e81febf38d3d",
"version" : "7.13.0",
"version_major" : 7
},
"trace" : {
"id" : "01c92d329ccd92478b83a238e3c01b4a"
},
"@timestamp" : "2021-07-19T18:07:31.953Z",
"ecs" : {
"version" : "1.8.0"
},
"service" : {
"node" : {
"name" : "CPX-GLE2FOXH0T3"
},
"environment" : "production",
"framework" : {
"name" : "ASP.NET Core",
"version" : "5.0.0.0"
},
"name" : "AppTest",
"runtime" : {
"name" : ".NET 5",
"version" : "5.0.5"
},
"language" : {
"name" : "C#"
},
"version" : "1.0.0"
},
"host" : {
"hostname" : "CPX-GLE2FOXH0T3",
"ip" : "172.25.0.1",
"name" : "CPX-GLE2FOXH0T3"
},
"client" : {
"ip" : "::1"
},
"http" : {
"request" : {
"headers" : {
"Cookie" : [
".AspNetCore.Antiforgery.2p9HK2lhxuY=CfDJ8Dx5pi93XlpAlBM6ht9NsLqRNgukbGGnh78TPr3xX0Z_y_FJlWrdLtle0U6p_8j675OuNHXMLqXvnbxdoBN7kcCl4XJrbJxXghxxnOD0cG36SGXjN544e9evy3i9_B1gQHMkqcTTSXkRZ1nSbHF5IzQ,.AspNetCore.Antiforgery.xEn_h8jTrzk=CfDJ8Dx5pi93XlpAlBM6ht9NsLpqaVgX_9OZKpxiQrooy-T3QRS3IJYkpHcsZ1bjWDNbaMNKRoVgAM_42kZLLojP3w9roVqV9CeOmrfH4W2SLXMZzvDHrpOLfgAGABkeTmIJp5BkuBMhCHg7zKgerE72zT8,.AspNetCore.Antiforgery.pWK10f4-nSU=CfDJ8Dx5pi93XlpAlBM6ht9NsLqE4evscYCvbn8gHWTG2rGBL4PZp77mBFUM0oHV8i3kJoamaj2sYos_ImQ6zFy_fDH-RQj02BT4ZQZAibdknXhEzidBJLmdqI3ZppITONjVz_ZVVjKkMTfnHRramkabzhk,.AspNetCore.Antiforgery.tphuApw-2rM=CfDJ8OKn5Y98jFhFj5OjgFbFoJXDU5pnA7XDUkNggYzhtLK3dCxde5OXjIUuU4zDm_6cJ9b3JK6A0k1i9APKs69jGU8_JmoAqDSi3cgnbLEV7xVEThEhMCdaHifsIXyw7rYZMdVBS5403oZfEUOKvpgviM0,.AspNetCore.Antiforgery.ohccJll1qP4=CfDJ8OKn5Y98jFhFj5OjgFbFoJWjauoWueKtAM1D9QkX575_VDMquYoyTciU3uUxyvwr6EW1fFKtdRjgtQoJUy1QthYYbrwNdJBggB7UfOKRpRgim4cOIL34WJ-0i7ayWW7_fliuPcJyo11XnMaiiUa-f3s,.AspNetCore.Antiforgery.u6V5pe-jSMU=CfDJ8OKn5Y98jFhFj5OjgFbFoJVYEagWGJ9Dfp0_g5dY-WMRwCLIzYQzCswRVRR79ELtrEMZQ58Qls-sMebGfgqUlRo-do0_Tu2kFnEL01_9l6Hv2eFSqx2i29ZDTFBSimXSIeKp6WOKJyvHmxp2S1270Uw,.AspNetCore.Antiforgery.nLROWordfdI=CfDJ8OKn5Y98jFhFj5OjgFbFoJUuL5nrnLNA0QoYDQAWAKeFtnvyETdbNan-EGYuOzwJUkD3qwSxXUa-wHnQ94JWUVfztTs-4tZXbGgjsQ0sYZ6M866kvAj1gLfp1-_BpkIk6JyXytRhb0C7w_eddqlxaZA"
],
"Sec-Ch-Ua" : [
"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\""
],
"Accept" : [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
],
"User-Agent" : [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
],
"Sec-Fetch-Dest" : [
"document"
],
"Sec-Fetch-Site" : [
"none"
],
"Host" : [
"localhost:5001"
],
"Accept-Encoding" : [
"gzip, deflate, br"
],
"Sec-Fetch-Mode" : [
"navigate"
],
":method" : [
"GET"
],
"Upgrade-Insecure-Requests" : [
"1"
],
":scheme" : [
"https"
],
":path" : [
"/"
],
"Sec-Fetch-User" : [
"?1"
],
"Accept-Language" : [
"en-US,en;q=0.9"
],
"Sec-Ch-Ua-Mobile" : [
"?0"
],
":authority" : [
"localhost:5001"
]
},
"method" : "GET",
"socket" : {
"encrypted" : true,
"remote_address" : "::1"
},
"body" : {
"original" : "[REDACTED]"
}
},
"response" : {
"headers" : {
"Server" : [
"Kestrel"
],
"Date" : [
"Mon, 19 Jul 2021 18:07:31 GMT"
],
"Content-Type" : [
"text/html; charset=utf-8"
]
},
"status_code" : 200,
"finished" : true
},
"version" : "2"
},
"event" : {
"ingested" : "2021-07-19T18:07:38.337682900Z",
"outcome" : "success"
},
"transaction" : {
"result" : "HTTP 2xx",
"duration" : {
"us" : 663305
},
"name" : "GET Home/Index",
"span_count" : {
"dropped" : 0,
"started" : 0
},
"id" : "0bd2bd7f8b94944e",
"type" : "request",
"sampled" : true
},
"user_agent" : {
"original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
"os" : {
"name" : "Windows",
"version" : "10",
"full" : "Windows 10"
},
"name" : "Chrome",
"device" : {
"name" : "Other"
},
"version" : "91.0.4472.124"
},
"timestamp" : {
"us" : 1626718051953315
}
}
},
{
"_index" : "apm-7.13.0-transaction-000002",
"_type" : "_doc",
"_id" : "3SXzv3oB2DAWJifJt8jM",
"_score" : 1.0,
"_source" : {
"agent" : {
"name" : "dotnet",
"version" : "1.11.0+a4eb5008af64ca52fe6973c18810c0c8c7a99554"
},
"source" : {
"ip" : "::1"
},
"processor" : {
"name" : "transaction",
"event" : "transaction"
},
"url" : {
"path" : "/Second/SecIndex",
"original" : "https://localhost:5001/Second/SecIndex",
"scheme" : "HTTP",
"domain" : "localhost",
"full" : "https://localhost:5001/Second/SecIndex"
},
"observer" : {
"hostname" : "ef75b7ae7c69",
"id" : "de63825b-0f59-4df0-aea9-4a538997f6e6",
"ephemeral_id" : "d050562a-8d75-4209-b7d0-e81febf38d3d",
"type" : "apm-server",
"version" : "7.13.0",
"version_major" : 7
},
"trace" : {
"id" : "92c232e28fe661468065148b329eeed3"
},
"@timestamp" : "2021-07-19T18:07:35.443Z",
"ecs" : {
"version" : "1.8.0"
},
"service" : {
"node" : {
"name" : "CPX-GLE2FOXH0T3"
},
"environment" : "production",
"framework" : {
"name" : "ASP.NET Core",
"version" : "5.0.0.0"
},
"name" : "AppTest",
"runtime" : {
"name" : ".NET 5",
"version" : "5.0.5"
},
"language" : {
"name" : "C#"
},
"version" : "1.0.0"
},
"host" : {
"hostname" : "CPX-GLE2FOXH0T3",
"ip" : "172.25.0.1",
"name" : "CPX-GLE2FOXH0T3"
},
"http" : {
"request" : {
"headers" : {
"Cookie" : [
".AspNetCore.Antiforgery.2p9HK2lhxuY=CfDJ8Dx5pi93XlpAlBM6ht9NsLqRNgukbGGnh78TPr3xX0Z_y_FJlWrdLtle0U6p_8j675OuNHXMLqXvnbxdoBN7kcCl4XJrbJxXghxxnOD0cG36SGXjN544e9evy3i9_B1gQHMkqcTTSXkRZ1nSbHF5IzQ,.AspNetCore.Antiforgery.xEn_h8jTrzk=CfDJ8Dx5pi93XlpAlBM6ht9NsLpqaVgX_9OZKpxiQrooy-T3QRS3IJYkpHcsZ1bjWDNbaMNKRoVgAM_42kZLLojP3w9roVqV9CeOmrfH4W2SLXMZzvDHrpOLfgAGABkeTmIJp5BkuBMhCHg7zKgerE72zT8,.AspNetCore.Antiforgery.pWK10f4-nSU=CfDJ8Dx5pi93XlpAlBM6ht9NsLqE4evscYCvbn8gHWTG2rGBL4PZp77mBFUM0oHV8i3kJoamaj2sYos_ImQ6zFy_fDH-RQj02BT4ZQZAibdknXhEzidBJLmdqI3ZppITONjVz_ZVVjKkMTfnHRramkabzhk,.AspNetCore.Antiforgery.tphuApw-2rM=CfDJ8OKn5Y98jFhFj5OjgFbFoJXDU5pnA7XDUkNggYzhtLK3dCxde5OXjIUuU4zDm_6cJ9b3JK6A0k1i9APKs69jGU8_JmoAqDSi3cgnbLEV7xVEThEhMCdaHifsIXyw7rYZMdVBS5403oZfEUOKvpgviM0,.AspNetCore.Antiforgery.ohccJll1qP4=CfDJ8OKn5Y98jFhFj5OjgFbFoJWjauoWueKtAM1D9QkX575_VDMquYoyTciU3uUxyvwr6EW1fFKtdRjgtQoJUy1QthYYbrwNdJBggB7UfOKRpRgim4cOIL34WJ-0i7ayWW7_fliuPcJyo11XnMaiiUa-f3s,.AspNetCore.Antiforgery.u6V5pe-jSMU=CfDJ8OKn5Y98jFhFj5OjgFbFoJVYEagWGJ9Dfp0_g5dY-WMRwCLIzYQzCswRVRR79ELtrEMZQ58Qls-sMebGfgqUlRo-do0_Tu2kFnEL01_9l6Hv2eFSqx2i29ZDTFBSimXSIeKp6WOKJyvHmxp2S1270Uw,.AspNetCore.Antiforgery.nLROWordfdI=CfDJ8OKn5Y98jFhFj5OjgFbFoJUuL5nrnLNA0QoYDQAWAKeFtnvyETdbNan-EGYuOzwJUkD3qwSxXUa-wHnQ94JWUVfztTs-4tZXbGgjsQ0sYZ6M866kvAj1gLfp1-_BpkIk6JyXytRhb0C7w_eddqlxaZA"
],
"Sec-Ch-Ua" : [
"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\""
],
"Accept" : [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
],
"User-Agent" : [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
],
"Referer" : [
"https://localhost:5001/"
],
"Sec-Fetch-Site" : [
"same-origin"
],
"Sec-Fetch-Dest" : [
"document"
],
"Host" : [
"localhost:5001"
],
"Accept-Encoding" : [
"gzip, deflate, br"
],
"Sec-Fetch-Mode" : [
"navigate"
],
":method" : [
"GET"
],
"Upgrade-Insecure-Requests" : [
"1"
],
":scheme" : [
"https"
],
":path" : [
"/Second/SecIndex"
],
"Sec-Fetch-User" : [
"?1"
],
"Accept-Language" : [
"en-US,en;q=0.9"
],
"Sec-Ch-Ua-Mobile" : [
"?0"
],
":authority" : [
"localhost:5001"
]
},
"method" : "GET",
"socket" : {
"encrypted" : true,
"remote_address" : "::1"
},
"body" : {
"original" : "[REDACTED]"
}
},
"response" : {
"headers" : {
"Cache-Control" : [
"no-cache, no-store"
],
"Server" : [
"Kestrel"
],
"Pragma" : [
"no-cache"
],
"Date" : [
"Mon, 19 Jul 2021 18:07:34 GMT"
],
"Content-Type" : [
"text/html; charset=utf-8"
]
},
"status_code" : 200,
"finished" : true
},
"version" : "2"
},
"client" : {
"ip" : "::1"
},
"event" : {
"ingested" : "2021-07-19T18:07:38.439690700Z",
"outcome" : "success"
},
"transaction" : {
"duration" : {
"us" : 64185
},
"result" : "HTTP 2xx",
"name" : "GET Second/SecIndex",
"span_count" : {
"dropped" : 0,
"started" : 0
},
"id" : "bc217d42a1806b40",
"type" : "request",
"sampled" : true
},
"user_agent" : {
"original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
"os" : {
"name" : "Windows",
"version" : "10",
"full" : "Windows 10"
},
"name" : "Chrome",
"device" : {
"name" : "Other"
},
"version" : "91.0.4472.124"
},
"timestamp" : {
"us" : 1626718055443632
}
}
},
However, what I cannot seem to determine is if there is a unique ID that will act as the parent ID /session ID that can correlate all the logs in a single activity. At first, I noticed that inside of the observer{} scope, there was an id and a ephemeral_id and when I searched for either of those id's from the result that the query dsl cmd gave me, it returned a total of 5 hits (for all of the 5 transactions made). But when I ran the query dsl cmd again but with a different timestamp I still had the same id and ephemeral_id so I knew that was not the ID that uniquely identifies the logs per a single activity. Is there a unique ID that uniquely identifies from the start to finish of an activity?
Transactions can be considered the root span for a trace, where a sampled transaction might have a collection of spans associated with it. These associated spans are stored in separate indices with the index pattern apm-*-span where the _source.transaction.id is the id of the parent transaction.
A distributed trace can span multiple services and hence multiple transactions. Transactions in a distributed trace will share a unique _source.trace.id which uniquely identifies the distributed trace through a system.