Search code examples
azure-data-explorerkqlkusto-explorer

Ingest multiple files from storage using Kusto

I exported about 60 millions rows using the .export command to a storage as my KQL script below:

.export async to json (h@"https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20;storage-account-access-key") <|
contosoEntity

I removed sentity parts from my query, but the exportation works fine. It exported 192 file(s), total of 62374652 rows.

I also need to import into another Kusto cluster, how can I use the .ingest command from storage to ingest all .json files from a specific folder from storage?

If there any other better solution to ingest back to another Kusto cluster, I will be glad to know.

What I tried and failed:

1-

.ingest into table VmCounterFiveMinuteRoleInstanceCentralBondTable_2 (h@'https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20/*.json;storage-account-access-key')

2-

.ingest into table VmCounterFiveMinuteRoleInstanceCentralBondTable_2 (h@'https://contoso.blob.core.windows.net/oversub/adx/VmCounterFiveMinutes/2023-05-20/*.ext;storage-account-access-key')
    with (format='json')
Solution

  • you can't specify a wildcarded (*) path when running the (unrecommended) direct ingestion command (.ingest).

    you could use LightIngest or the Ingestion wizard to ingest the files you've exported (with filters on blob name prefix, if required)