Search code examples
azurevirtual-machineubuntu-16.04azure-storageazure-disk

Unknown Disk Read Bytes and Disk Operations at Azure Ubuntu VM service after that VM is not responding


Each week nearly two or three times, azure vm stops the responding. When it happens, Disk Read Bytes are dramatically increasing i dont know why. My application has no read disk function (only writes the logs). As you see in the figure, Read Bytes and Disk Operations are increasing for a moment that causes server to freeze. I have to do restart the VM and it takes nearly 15 minutes to be available again.

Azure Ubuntu VM Stats

i am using Azure Ubuntu 16.04 VM and Size is Standard B1s (1 vcpus, 1 GiB memory).

i have alredy checked these conditions

  • possible memory leaks by my application
  • observed all source codes of application whether disk read function exists or not (i dont have any disk ready operation at my application)
  • i have changed default virtual ram size at ubuntu %60 to %1
  • i am using docker to run my application and i am using only 1 instance of docker image at vm
  • i removed all vm and i created it again (including all resources i.e. disk/network etc.)

i want to know why this is happening and how can i investigate what are cause this problem. I haven't seen any suspicious running process at ubuntu when it is happening.


Solution

  • This issue is solved by Azure Technical Support Engineers (Thanks to Micah_MSFT)

    That seems Azure VM is going to out of memory. When it happens, VM is acting unsteady, so Disk Operations is rapidly increasing and VM is freezing (they are investigating the issue's details and reported to Linux Technicians).

    As clarified by Azure Technical Support Engineer, There is a feature named Swap Space which helps the saver to reserve storage space as processing memory.

    Here is the configuration

    1. Open the file located under /etc/waagent.conf find these variables and modify them accordingly:
    # Format if unformatted. If 'n', resource disk will not be mounted.​
    ResourceDisk.Format=n​   /// change this parameter to "y"
    
    # Create and use swapfile on resource disk.​
    ResourceDisk.EnableSwap=n​   /// change this parameter to "y"
    ​
    # Size of the swapfile.​
    ResourceDisk.SwapSizeMB=0​   /// change this parameter to "1024"
    
    1. Restart the service to apply the changes

    sudo systemctl restart walinuxagent.service

    That's it

    In my opinion, unnecessary increase in disk usage as a result of Out of Memory limit will cause Azure VMs performance to decrease.