Search code examples
azure-data-explorerdata-ingestion

How to ingest data from multiple resources with different schema


I need to centralize the logs from multiples resource with different schema

For example

Schema in source1: Timestamp, errorcode, errordescription

Schema in source2: Timestamp, eventId, event description

Table is created as

.create table TraceLog (Timestamp:datetime, errorcode:string, errordescription:string, eventId:string, eventdescription:string)

For ingest data from source1, I cam simple use

.ingest inline into Table TraceLog <| 2020-05-01, "error code", "test error description"

For data from soruce2, except

.ingest inline into Table TraceLog <| 2020-05-01, "", "", "test event ID", "test event description"

any other way to ingest data without specify column in source1?


Solution

  • The simplest option would be using ingestion mappings per different source:

    https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings

    In case the data requires additional transformation (e.g. parsing): you can utilize Update Policy:

    https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/updatepolicy