Search code examples
pythonazureazure-functionsmonitor

Python: How to check if Azure Function is still running and has queue

I know there is this post (how to check if Azure Function is still running or not) which has a similar question. It doesn't really help.

I have a function that receives inputs during a period of time in batches (e.g: 1000 batches of n samples each). Batches don't arrive all at the same time. The function processes each batch and writes the output to a blob in a blobstoragecontainer.

My question is how can I know that the results have all been written into the blob storage so I can trigger a download?

I was trying to look into azure-mgmt-monitor to check if I could monitor the number of requests/function calls over the last minute and put that into a while loop until some metrics (might be others) are 0. I can call some metrics over a timespan with some aggregation, but for some reason I keep getting all values as 0 (when I know there were calls).

Below some code (azure_mngr.monitor is one instance of azure.mgmt.monitor.MonitorManagementClient:

start_date = datetime.datetime(2020, 6, 2, 10, 0, 0)
end_date = datetime.datetime(2020, 6, 2, 11, 0, 0)
azure_mngr.monitor.metrics.list(azure_function_id, metricnames='Requests,RequestsInApplicationQueue,FunctionExecutionCount', interval='PT1M', timespan=f'{start_date}/{end_date}', aggregation='Count,Average,Maximum,Minimum')

Result

for metric in a.value:
    for datapoint in metric.timeseries[0].data:
        print(f'{metric.name.value} | {datapoint.time_stamp} | {datapoint.count}')

Requests | 2020-06-02 10:00:00+00:00 | 0.0
Requests | 2020-06-02 10:15:00+00:00 | 0.0
Requests | 2020-06-02 10:30:00+00:00 | 0.0
Requests | 2020-06-02 10:45:00+00:00 | 0.0
RequestsInApplicationQueue | 2020-06-02 10:00:00+00:00 | 0.0
RequestsInApplicationQueue | 2020-06-02 10:15:00+00:00 | 0.0
RequestsInApplicationQueue | 2020-06-02 10:30:00+00:00 | 0.0
RequestsInApplicationQueue | 2020-06-02 10:45:00+00:00 | 0.0
FunctionExecutionCount | 2020-06-02 10:00:00+00:00 | 0.0
FunctionExecutionCount | 2020-06-02 10:15:00+00:00 | 0.0
FunctionExecutionCount | 2020-06-02 10:30:00+00:00 | 0.0
FunctionExecutionCount | 2020-06-02 10:45:00+00:00 | 0.0

And graph with requests count at that period (from insights resource) enter image description here

My wildest guess tells me that the id I should be passing might not be the azure function one but maybe another... I also don't have any other idea on how to do this. I have also been looking at the azure-mgmt-applicationinsights, but it is even more obscure than the monitor one...

Solution

  • So after some days I found a way to read from azure insights, though I didn't stick with that solution.

    INSIGHTS METHOD

    For anyone interested, to read metrics from the insights you need to use azure-applicationinsights package (not azure-mgmt-applicationinsights). Then you instantiate a ApplicationInsightsDataClient with your azure credentials and you can send queries as follow:

    from azure.applicationinsights import ApplicationInsightsDataClient

client = ApplicationInsightsDataClient(<credentials>)
metric = client.metrics.get(<application_id>, <metric>, custom_headers=<custom_headers>, **<other_kwargs>)

    Now, the tricky part is this "application_id" and "custom_headers". For the first you can retrieve it from azure, inside the insights resource on the API Access Keys tab.

    For the custom headers, you will need to specify your token to one API_key you need to create in the insights resource (same place as above). Format should be:

    custom_headers = {'x-api-key': <api_key>}

    To know which metrics you can get from the insights resource you can

    available_metrics = client.metrics.get_metadata(<application_id>, custom_headers=custom_headers)

    Additionally, if you have your deployment in a CI/CD pipeline, you can also automate the retrieval of the ApplicationID and creation of api_key.

    To get the application id on gitlab pipeline logs I just created and output for it in terraform (check terraform documentation on outputs) For the api_key creation, you then need to use the azure-mgmt-applicationinsights package and instantiate a ApplicationInsightsManagementClient (not sure of this name right now)

    Pass credentials and in "api_keys" attribute use the "create" method. This might be a bit tricky, since you need to pass some "linked_read_properties".

    What I suggest is create it in azure portal first, read it in python, check the attributes you need and then try to create it through python.

    As for the solution I end up to stick with.

    I made my azure function that was writing the results into the blob_storage to also write a "metadata" key/value.

    If there is some batch yet to run, it will write it as False. If it is the last batch will change it to True.

    Then I just read the blob properties in python until the value is True.

    Hope this can help anyone out there with similar issue ;)