How to delete logs from Azure using custom conditions?

Question

I am trying to find a way to delete logs from Azure using some custom conditions(automatically or manually it doesn't matter right now) for e.g:

Say I want to delete all logs from Azure except the exceptions that occurred in the last 3 days with a customEvent name of 'LogIn'.

Is there a way to do this in Azure ?

Thanks in advance!

[My Context]

I am using Application Insights with Asp.Net Core 3.1 and Application Insights that I log with Serilog Structured Logging.

I send my log as events then do custom queries for app monitoring and alerts based on that data.

A few examples of Serilog calls:

  _logger.Debug("{randomValue} {temperature}", rng, temperature);
  _logger.Information("{_class_} {_function_}", nameof(UserService), nameof(IsLoggedIn));
  _logger.Error(ex, "{_function_}", nameof(LogIn));

Answer 1

To delete logs from application insights, the only way is to use the purge rest api. But it has some limitations.

Remember that you should delete the logs from each table at each time. For example, you should delete logs from traces table / requests table etc. respectively. All the tables are listed below:

1.Here is an example of delete logs from traces table by using purge rest api. Nav to the purge api page, click the try it button(the button is green) -> then log in, and fill in all the necessary information -> and in the body field, set the filter as below:

Note that, after it returns successfully, scroll down to the bottom of the page, there is "operationId". Please write it down, and you can use it to check the purge status via Get Purge Status.

2.Now, you can use the same method to delete logs from your exceptions table. And as per your need, you should do more purge operations. First, as mentioned above, delete all the exceptions logs by specifying a timestamp which is 3 days ago. Second, you can delete the logs in the latest 3 days which event name is not "LogIn" as per the filter as below:

{
        "table": "exceptions",
        "filters":  [
          {
            "column": "the column name",
            "operator": "==",
            "value": "not LogIn"         
          }
        ]   
}

The limitation here is that the api now not supports the operator !=, so you need to delete the logs whose customEvent name is not "LogIn" use filter above. The full supported operators are here.

3.The last thing is that, due to some reasons, even if we did the purge, the logs are still kept about several days(7 days or 3 days) in application insights. Please use the Get Purge Status to check the progress.