Search code examples
azureloggingazure-log-analyticsazure-storage-account

Analyse log files in Azure blob storage

I have on-prem Linux machines, we store their logs in our Azure storage account as blobs. Can we use Azure log analytics to collect these logs stored in the blob storage?

Solution

  • ​Yes, we can use Azure log analytics to collect the logs. There are 2 ways:

    WAY-1 Try following the below steps

    1. Fill in the required parameters and execute the script locally or in Azure Cloud Shell.
    2. This PowerShell script downloads the logs from Azure Storage.
    3. Convert the diagnostic logs into JSON format, as that is what the API expects.
    4. Load custom data into Log Analytics using the HTTP Data Collector API.

    For more information, you can refer to this Document

    WAY-2 There is a direct process where you can install a log analytics agent in the virtual machines through Azure Monitor.

    Azure Monitor collects the data directly from your physical or virtual Linux computers in your environment into a Log Analytics workspace for detailed analysis and correlation using the azure log analytics agents.

    Steps to follow in order to collect data from data center using Azure monitor STEP - 1: Install the log analytics agent for Linux

    1. Enter the log analytics workspace id and primary key in the following command.
        wget https://raw.githubusercontent.com/Microsoft/OMS-Agent-for-Linux/master/installer/scripts/onboard_agent.sh && sh onboard_agent.sh -w <YOUR WORKSPACE ID> -s <YOUR WORKSPACE PRIMARY KEY>

    enter image description here 2. Enter the log analytics workspace id in the following command and restart the agent

    sudo /opt/microsoft/omsagent/bin/service_control restart [<workspace id>]

    STEP - 2: Collect the event and performance data

    1. Azure portal --> Log Analytics --> Log Analytics workspace --> Advance settings --> Data --> Syslog --> '+' sign.
    2. Uncheck the severities Info, Notice, Debug and then Apply. enter image description here
    3. Now go to Linux Performance counters and click Add Recommended Counters and click Apply. enter image description here

    STEP - 3: View Collected Data

    1. Log Analytics Workspace --> Logs (from the left pane)
    2. On the Logs query page, type Perf in the query editor and select Run.

    For more information, you can refer to this Blog.