Search code examples
azureazure-storage-account

What all logs/Metrics should be enabled as part of the Diagnostic settings enablement for Azure Storage Accounts

As part of a client requirement, I've been asked to set up central log repository for different Azure workloads including Storage accounts & databases. I see a default diagnostic setting in place but all of those are disabled. To enable these, we need to enable certain logs/metrics which will further be ingested into the workspace. Now I want to make a cost-effective & most accurate selection of the logs/metrics for storage accounts. Can someone with more profound knowledge into this domain enlighten me about it?

enter image description here Similarly for Postgre SQL & Cosmos DB databases too, I have to make such decision. Please help me with this.

Solution

  • Please check the below points and references in detail.

    Selection:

    You can select the logs for the operations that you want to Get all the details you wish for.selection depends on the requirement.

    1. A good practice is to go through your agents and monitoring settings and see exactly what you are logging. Capture logs which are important for your monitoring purpose.
    2. Choose the cheapest region to create and store your log analytics workspace.
    3. If you have very high volume of the log ingestion then it would be prudent to opt for azure commitment tier.
    4. In case you need to export the log analytics data, rather than exporting all the data, you can filter it and send only relevant log data

    Above things can significantly reduce your azure billing cost and help you to save money in using azure monitor effectively. Understand Azure Monitor and Log Analytics Pricing and Cost Optimization (azurelib.com)

    Storing:

    Log data can accumulate in your account over time which can increase the cost of storage.

    1. If you need log data for only a small period of time, you can reduce your costs by modifying the log data retention period to less days.
    2. Use lifecycle policy to move data between access tiers.
    3. Data ingested into Log Analytics workspace can be retained at no additional charge(free) up to the first 31 days.

    See Design considerations and change the data retention if not needed more than that. See Monitoring Azure Blob Storage | Microsoft Docs.

    1. Storage Insights is a dashboard on top of Azure Storage metrics and logs. You can use Storage Insights to examine the transaction volume and used capacity of all your accounts. That information can help you decide which accounts you might want to retire.

    Analyze:

    1. Analyze the used capacity and monitor the use of the container.
    2. you can consider reducing the total cost by exporting logs to storage account, and then using a serverless query solution on top of log data.See blob storage monitoring/optimize cost for infrequent queries
    3. Organize data into access tiers.Log Analytics has Commitment Tiers, which can save you as much as 30 percent compared to the Pay-As-You-Go price.
    4. You should periodically review this information to determine if you can reduce your charges by moving to another tier

    References:

    1. Plan and manage costs for Azure Blob Storage | Microsoft Docs
    2. Azure Monitor Logs pricing details - Azure Monitor | Microsoft Docs
    3. Azure Monitor Log Analytics too Expensive? Part 2 - Save Some Money | Thomas Stringer (trstringer.com)