Previously ingesting data into an Azure Log Analytics Log Table could be done without the need for a DCR (Data Collection Rule) or indeed a DCE (Data Collection Endpoint). In fact, this stack overflow post covers this powershell-script-for-azure-function-to-post-to-a-log-analytics.
Microsoft have changed how this works and so if one creates a modern Log Table and point the code detail in the linked article, the data is simply dropped. (The console output looks successful. However, there is no data ingested...simple KQL query against the log table confirms this)
The first problem being faced is to create a DCR that ingest data via the REST API and direct it to the requisite log table. In this instance there is no need for a DCE.
This is the code being used to create the DCR, which fails:
$DCRContent = @"
{
"location": "westeurope",
"properties": {
"dataSources": {
"logsIngestion": [
{
"name": "RestApiSource",
"streams": ["<Custom Table Name>"],
"settings": {
"endpoint": "https://<workspace ID>.ods.opinsights.azure.com/api/logs?api-version=2023-01-01",
"method": "POST",
"headers": {
"Content-Type": "application/json"
}
}
}
]
},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/<Subscription ID>/resourceGroups/<ResourceGroupName>/providers/Microsoft.OperationalInsights/workspaces/<Workspace Name>"
}
]
},
"transformations": [
{
"name": "Transformation1",
"inputs": ["RestApiSource"],
"outputs": ["<Custom Table Name>"],
"query": "SELECT * FROM RestApiSource"
}
]
}
}
"@
$ResourceGroupName = $Workspace.Rg
$DCRName = "<DCR Name>"
$Location = "westeurope"
New-AzResource -ResourceGroupName $ResourceGroupName -ResourceType "Microsoft.Insights/dataCollectionRules" -ResourceName $DCRName -Location $Location -Properties $DCRContent
This is the error message:
New-AzResource : InvalidPayload : Data collection rule is invalid
CorrelationId: 1fcde55d-6770-48c7-b649-822eca399203
At line:47 char:1
+ New-AzResource -ResourceGroupName $ResourceGroupName -ResourceType "M ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [New-AzResource], ErrorResponseMessageException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceCmdlet
Fundamentally this needs to be done with PowerShell ver >= 7. Also the account performing the action must have Monitoring Metric Publisher.
Initially I used the UI to create both table and DCR at the same time. This provided the JSON needed to understand how the ARM template for the DCR should be constructed.
As I need to create multiple tables I wanted to automated the creation of the table and the DCR. I already had a DCE in my environment, which is operating well within its limits, there was no need to create new one specifically for this. The data to be ingested would always be in an array whose elements are identical hash tables.
With this in mind I created a function that could process the array, working out data types of each value in the hash table. It then creates the Log Analytics table and the associated DCR based on this information. Data types were challenging for a couple of reasons, either not directly translating or requiring some transformation (the function handles this, but could be expanded).
Here is the function (preamble removed):
function New-AzLogAW-Table(){
param(
[parameter(Mandatory = $true, Position = 1)]
[PSObject]$Workspace,
[parameter(Mandatory = $true, Position = 2)]
[string]$TableName,
[parameter(Mandatory = $true, Position = 3)]
[PSObject]$Data,
[parameter(Mandatory = $true, Position = 4)]
[string]$Description,
[parameter(Mandatory = $false, Position = 5)]
[string]$RetentionInDays = 7,
[parameter(Mandatory = $false, Position = 6)]
[string]$TotalRetentionInDays = 12,
[parameter(Mandatory = $true, Position = 7)]
[string]$DCE_ID
)
$TableName += "_CL"
# Determine columns to be created and their data types.
$columns = @{}; $Data[0].PSObject.Properties | % {
if ($null -eq $_.Value) {
$type = "string"
if ($_.Name -match "^is") {$type = "boolean"}
if ($_.Name -match "time" -or $_.Name -match "date") {$type = "datetime"}
}
else {
$type = ($_.Value.GetType()).Name
if ($type -eq "String") {
$type = "string"
# Test if the value is of type Datetime and change it accordingly.
$retVal = $_.Value -as [Datetime]
if ($retVal) { $type = "datetime" }
else {
# Test if the value is of type Int32 and change it accordingly to Int.
$retVal = $_.Value -as [Int32]
if ($retVal) { $type = "int" }
}
}
if ($type -eq "Int32" ) { $type = "int" }
if ($type -eq "Boolean" ) { $type = "boolean" }
if ($type -eq "Int64" ) { $type = "long"}
if ($type -eq "DateTime" ) { $type = "datetime"}
}
$columns.Add($_.Name, $type)
}
$params = @{
ResourceGroupName = $Workspace.Rg
WorkspaceName = $Workspace.Name
TableName = $TableName
Description = $Description
RetentionInDays = $RetentionInDays
TotalRetentionInDays = $TotalRetentionInDays
Column = $columns
}
#return $columns
# Create Azure Log Analytics Workspace custom table.
try {
Log-Entry "Create Log Analytics table `"$TableName`". . ."
$ALAWtable = New-AzOperationalInsightsTable @params
Log-Entry "Successfully created." -Margin 2
Log-Entry " "
}
catch { Write-Error "Failed to create table [$TableName]."; return @($false,$false)}
# Create Data Collection Rule (DCR).
$DCR_ARM = '{"$schema":"https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion":"1.0.0.0","resources":[{"apiVersion":"2023-03-11",
"name":"dcr-mgmt-restAPI-<TableName>","type":"Microsoft.Insights/dataCollectionRules",
"location":"westeurope","kind":"Direct",
"properties":{"dataCollectionEndpointId":"<DCE_Prefix>/microsoft.insights/dataCollectionEndpoints/<DCE_ID>",
"streamDeclarations":{"Custom-<TableName>":{"columns":<Columns>}},"dataSources":{},
"destinations":{"logAnalytics":[{"workspaceResourceId":"<workspaceResourceId>",
"workspaceId":"<WorkspaceId>","name":"<WorkspaceId>"}]},"dataFlows":[{"streams":["Custom-<TableName>"],
"destinations":["<WorkspaceId>"],"transformKql":"source",
"outputStream":"Custom-<TableName>"}]}}]}'
# Update DCR ARM Template.
$DCR_ARM = $DCR_ARM.Replace("<TableName>",$tableName)
$DCR_ARM = $DCR_ARM.Replace("<WorkspaceId>",$Workspace.ID)
$DCR_ARM = $DCR_ARM.Replace("<workspaceResourceId>",$workspace.ResourceId)
if ($workspace.ResourceId -match "(^\/sub[\da-zA-Z\%-_ \/]*providers)") { $DCE_Prefix = $matches[1] }
else {Write-Error "Invalid Resource ID!"; @($ALAWtable,$false) }
$DCR_ARM = $DCR_ARM.Replace("<DCE_Prefix>",$DCE_Prefix)
$DCR_ARM = $DCR_ARM.Replace("<DCE_ID>",$DCE_ID)
$col = @()
foreach($r in $columns.GetEnumerator()) { $r2 = @{name=$r.Name; type=$r.Value}; $col += $r2 }
$Tbl_Columns = ($col | ConvertTo-Json) -replace '\s+', ''
$DCR_ARM = $DCR_ARM.Replace("<Columns>",$Tbl_Columns)
$DCR_ARM_obj = ConvertTo-Hashtable -InputObject ($DCR_ARM | ConvertFrom-Json)
# Deploy the ARM template Object
try {
Log-Entry "Creating DCR `"dcr-mgmt-restAPI-$tableName`". . ."
$DCR_retVal = New-AzResourceGroupDeployment -ResourceGroupName $Workspace.Rg -TemplateObject $DCR_ARM_obj -DeploymentDebugLogLevel All
}
catch { Write-Error "Failed to create DCR [dcr-mgmt-restAPI-$TableName].";$DCR_retVal = $false}
return @($ALAWtable,$DCR_retVal)
}
With this aspect of the problem solved attention turned to ingesting the data. For this I created the following function that uses a lot of what Microsoft provided in respect to doing this. It throttles the amount of data being sent to 500 records per upload.
Here's the function:
function Set-AzLogAW-Records() {
param(
[parameter(Mandatory = $true, Position = 1)]
[PSObject]$Workspace,
[parameter(Mandatory = $true, Position = 2)]
[string]$TableName,
[parameter(Mandatory = $true, Position = 3)]
[PSObject]$Data
)
$c = $Data.Count
Log-Entry " "
Log-Entry -Bar -
Log-Entry "[$c] records to upload to `"$TableName`"." -Justify C
Log-Entry -Bar -
Log-Entry " "
$Msg = "End of Upload."
# Get Data Collection Rules.
Log-Entry "Get DCR configuration for table `"$TableName`"..."
try { $dcr = Get-AzDataCollectionRule -ResourceGroupName $Workspace.Rg -ErrorAction Stop | ? { $_.StreamDeclaration.Keys -match $tableName } }
catch { Log-Entry "Failed to get DCR configuration." -Type error -Margin 2; return CloseFn $Msg $false }
Log-Entry "DCR Name.....: $($dcr.Name)" -Margin 2
Log-Entry "ImmutableID..: $($dcr.ImmutableId)" -Margin 2
Log-Entry "Stream Name..: $($dcr.StreamDeclaration.Keys)" -Margin 2
Log-Entry "End Point ID.: $($dcr.DataCollectionEndpointId)" -Margin 2
Log-Entry " "
# Get Data Collection Endpoint.
$dcr.DataCollectionEndpointId -match "dataCollectionEndpoints\/([\w\- ]*$)" | Out-Null
$dceName = $matches[1]
Log-Entry "Get DCE `"$dceName`" configuration..."
try { $dce = Get-AzDataCollectionEndpoint -ResourceGroupName $Workspace.Rg -Name $dceName -ErrorAction Stop }
catch { Log-Entry "Failed to get DCE configuration." -Type error -Margin 2; return CloseFn $Msg $false }
Log-Entry "Log Ingestion Endpoint: $($dce.LogIngestionEndpoint)" -Margin 2
Log-Entry " "
# Get Bearer Token.
$bearerToken = Get-BearerToken
# Upload data to Log Analytics table.
$headers = @{"Authorization" = "Bearer $bearerToken"; "Content-Type" = "application/json" };
$uri = "$($dce.LogIngestionEndpoint)/dataCollectionRules/$($dcr.ImmutableId)/streams/$($dcr.StreamDeclaration.Keys)?api-version=2023-01-01"
$rc = 0; $inc = 500 ; $ic = 0
Log-Entry -Bar -
Log-Entry "Upload to $tableName . . ."
Log-Entry -Bar -
Log-Entry " "
for ($rc = 0; $rc -le $Data.Count; $rc += $inc) {
if (($Data.Count - $rc) -lt $inc) { $inc = $Data.Count - $rc }
if ($Data.Count - $rc -lt 1) { break }
$to = $rc + $inc - 1
$body = $Data[$rc..$to] | ConvertTo-Json
try {
Log-Entry "Uploading [$rc..$to] records . . ."
$retVal = Invoke-RestMethod -Uri $uri -Method "Post" -Body $body -Headers $headers
Log-Entry "Upload completed successfully." -Margin 2
}
catch { Log-Entry "Failed to upload records [$_]" -Type error -Margin 2; return CloseFn $Msg $false }
}
return CloseFn $Msg $true
}
There are few other functions in here that can easily be replaced, like Log-Entry for Write-Host.
Hope this helps anyone else facing this challenge.