Search code examples
azure-devopsazure-pipelines

How to get Azure DevOps Pipeline step logs accessible from another future step?

I am trying to mess around with Azure DevOps pipelines, specifically I'm trying to add a python script that uses an AI LLM and a file it receives to analyze it and offer a humane description of what issues there is and how to address them. The goal of this is to have a step where, upon a failure during the pipeline, said logs should be collected into a txt file or similar, something that the python script can read, so that I can then use it to analyze what happened. I am a bit inexperienced in Azure DevOps and Pipelines in general, so I am struggling to find how to access logs from a previous, failed step, or even how to store them in the agent temp somehow.

Any ideas?

I tried adding a try catch on my step, but it didn't really work. I tried searching the pipeline's working directory for logs and files containing certain words but I found nothing.

I was expecting that every task would have stored something somewhere, but apparently I was wrong.

Solution

  • I am struggling to find how to access logs from a previous, failed step, or even how to store them in the agent temp somehow.

    You can add a PowerShell task at the end of the pipeline, which uses the REST API Timeline - Get to get the log of the previous failed step. In the response of the API, we can get the log URL for each step. We can get the log of the failed step and store it in the agent temp you need. Then you can add your python script task to read the downloaded log.

    Here is the sample pipeline YAML:

    trigger:
- none

pool:
  vmImage: windows-latest

steps:
- script: |
    FAIL
  displayName: "task1"
- script: |
    echo "task2"
  displayName: "task2"

- task: PowerShell@2
  condition: always()
  displayName: "download logs of previous failed step"
  inputs:
    targetType: 'inline'
    script: |
      $orgname=""
      $projectname=""
      
      $url = "https://dev.azure.com/$orgname/$projectname/_apis/build/builds/$(Build.BuildId)/timeline?api-version=7.1"
      $TimelineResponse = Invoke-RestMethod -Uri $url -Headers @{"Authorization" = "Bearer $(System.AccessToken)"}
      # Print TimelineResponse in the task log
      # $TimelineResponse  | ConvertTo-Json -Depth 10
      # Save the Timeline Response to $(Build.ArtifactStagingDirectory)/TimelineResponse.json
      # $TimelineResponse  | ConvertTo-Json -Depth 10 >> $(Build.ArtifactStagingDirectory)/TimelineResponse.json
      
      # Loop through each record and download the logs for failed step to $(Build.ArtifactStagingDirectory) folder
      foreach ($record in $TimelineResponse.records) 
      {
          if ($record.type -eq "Task" -and $record.result -eq "failed" -and $record.log -and $record.log.url) {
              $logUrl = $record.log.url
              $taskName = $record.name
              $logFile = Join-Path -Path "$(Build.ArtifactStagingDirectory)" -ChildPath "$taskName.log"
              
              # Download the log file of failed step 
              $log =  Invoke-WebRequest -Uri $logUrl -OutFile $logFile  -Headers @{"Authorization" = "Bearer $(System.AccessToken)"}
              Write-Output "Downloaded log from $logUrl to $logFile"
          }
      }


- publish:  $(Build.ArtifactStagingDirectory)
  condition: always()

    In this sample, task1 will fail. So, I added the condition: always() to the PowerShell task to make it run even if a previous task fails. The publish task is used to download the log so that I can check if the log file is correct.

    Test result:

    result