We need to initially import and transform large amounts of data in Azure Data Explorer.
The transformation consists of several .set operations containing queries representing transformational steps.
When i run those queries, they exeed certain limits ADX imposes to protect the cluster.
The error message is
aggregation over string column exceeded the memory budget of 8GB during evaluation
I know i can override those default limits for memory consumption per iterator and per node but this seems to only work for queries without ingestion commands.
When i want to run
set max_memory_consumption_per_query_per_node=68719476736;
.set async DestinationTable <| SourceTable | ...
ADX complains with
The incomplete fragment is unexpected.
Is there a way to temporarily increase those query limits for .set-ingest operations as well?
The limit you're hitting isn't configurable using the options you included above.
The best approach would be optimizing the query/command you're running - you can start with query best practices and considering splitting the single .set-or-append command into multiple ones (see "Remarks" here).
If these still don't help, I would recommend that you include the full command text here for further advice.