Search code examples
azure-data-explorerdata-ingestion

IngestFromStreamAsync method does not work


I manage to ingest data successfully using below code

var kcsbDM = new KustoConnectionStringBuilder(
    "https://test123.southeastasia.kusto.windows.net", 
    "testdb")
  .WithAadApplicationTokenAuthentication(acquireTokenTask.AccessToken);

using (var ingestClient = KustoIngestFactory.CreateDirectIngestClient(kcsbDM))            
{

    var ingestProps = new KustoQueuedIngestionProperties("testdb", "TraceLog");
    ingestProps.ReportLevel = IngestionReportLevel.FailuresOnly;
    ingestProps.ReportMethod = IngestionReportMethod.Queue;
    ingestProps.Format = DataSourceFormat.json;
 
    //generate datastream and columnmapping

    ingestProps.IngestionMapping = new IngestionMapping() { 
      IngestionMappings = columnMappings };
    var ingestionResult = ingestClient.IngestFromStream(memStream, ingestProps);
}

when I try to use QueuedClient and IngestFromStreamAsync, the code is executed successfully but no any data is ingested into database even after 30 minutes

var kcsbDM = new KustoConnectionStringBuilder(
    "https://ingest-test123.southeastasia.kusto.windows.net", 
    "testdb")
  .WithAadApplicationTokenAuthentication(acquireTokenTask.AccessToken);

using (var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(kcsbDM))          
{

    var ingestProps = new KustoQueuedIngestionProperties("testdb", "TraceLog");
    ingestProps.ReportLevel = IngestionReportLevel.FailuresOnly;
    ingestProps.ReportMethod = IngestionReportMethod.Queue;
    ingestProps.Format = DataSourceFormat.json;
 
    //generate datastream and columnmapping

    ingestProps.IngestionMapping = new IngestionMapping() { 
      IngestionMappings = columnMappings };
    var ingestionResult = ingestClient.IngestFromStreamAsync(memStream, ingestProps);
}

Solution

  • I find the reason finally, need to enable stream ingestion in the table:

    .alter table TraceLog policy streamingingestion enable
    

    See the Azure documentation for details.