AzureDevops build pipeline gets cancelled due to time out during donet build command
I have a build pipeline that build my .net 5 solution projects in Azure Devops. It was working fine till now, but has recently started failing in the build stage due to time out. The build command gets stuck with below logs:
2024-03-18T10:53:27.4401495Z ##[section]Starting: Buid solution
2024-03-18T10:53:27.4525992Z ==============================================================================
2024-03-18T10:53:27.4526134Z Task : .NET Core
2024-03-18T10:53:27.4526208Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command
2024-03-18T10:53:27.4526329Z Version : 2.235.1
2024-03-18T10:53:27.4526385Z Author : Microsoft Corporation
2024-03-18T10:53:27.4526457Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
2024-03-18T10:53:27.4526581Z ==============================================================================
2024-03-18T10:53:27.8552527Z [command]C:\Windows\system32\chcp.com 65001
2024-03-18T10:53:27.8655298Z Active code page: 65001
2024-03-18T10:53:27.8684055Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
2024-03-18T10:53:28.3062787Z [command]C:\hostedtoolcache\windows\dotnet\dotnet.exe build D:\a\1\s\src\MES.CloudApplication.sln "-dl:CentralLogger,\"D:\a\_tasks\DotNetCoreCLI_5541a522-603c-47ad-91fc-a4b1d163081b\2.235.1\dotnet-build-helpers\Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll\"*ForwardingLogger,\"D:\a\_tasks\DotNetCoreCLI_5541a522-603c-47ad-91fc-a4b1d163081b\2.235.1\dotnet-build-helpers\Microsoft.TeamFoundation.DistributedTask.MSBuild.Logger.dll\""
2024-03-18T10:53:28.5921283Z Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET
2024-03-18T10:53:28.5921873Z Copyright (C) Microsoft Corporation. All rights reserved.
2024-03-18T10:53:28.5922177Z
2024-03-18T10:53:29.2303494Z Determining projects to restore...
2024-03-18T10:53:30.1778985Z Restored D:\a\1\s\tests\MES.CloudApplication.Infrastructure.Tests\MES.CloudApplication.Infrastructure.Tests.csproj (in 624 ms).
2024-03-18T10:53:30.1779760Z Restored D:\a\1\s\tests\MES.CloudApplication.Application.Tests\MES.CloudApplication.Application.Tests.csproj (in 512 ms).
2024-03-18T10:53:30.2894980Z Restored D:\a\1\s\src\MES.CloudApplication.Workflow\MES.CloudApplication.Workflow.csproj (in 83 ms).
2024-03-18T10:53:30.3039944Z Restored D:\a\1\s\tests\MES.CloudApplication.Api.Tests\MES.CloudApplication.Api.Tests.csproj (in 99 ms).
2024-03-18T10:53:30.5759744Z Restored D:\a\1\s\src\MES.CloudApplication.Printing\MES.CloudApplication.Printing.csproj (in 183 ms).
2024-03-18T10:53:30.5760975Z Restored D:\a\1\s\src\MES.CloudApplication.Printing.Tests\MES.CloudApplication.Printing.Tests.csproj (in 235 ms).
2024-03-18T10:53:30.7334699Z Restored D:\a\1\s\src\MES.CloudApplication.Integration\MES.CloudApplication.Integration.csproj (in 216 ms).
2024-03-18T10:53:30.7335992Z Restored D:\a\1\s\src\MES.CloudApplication.Infrastructure\MES.CloudApplication.Infrastructure.csproj (in 54 ms).
2024-03-18T10:53:30.9119467Z Restored D:\a\1\s\src\MES.CloudApplication.Domain\MES.CloudApplication.Domain.csproj (in 12 ms).
2024-03-18T10:53:30.9120408Z Restored D:\a\1\s\src\MES.CloudApplication.Application\MES.CloudApplication.Application.csproj (in 175 ms).
2024-03-18T10:53:30.9720180Z Restored D:\a\1\s\src\MES.CloudApplication.Api\MES.CloudApplication.Api.csproj (in 218 ms).
2024-03-18T10:53:34.4069633Z MES.CloudApplication.Api.Tests -> D:\a\1\s\tests\MES.CloudApplication.Api.Tests\bin\Debug\net5.0\MES.CloudApplication.Api.Tests.dll
2024-03-18T10:53:34.5178908Z MES.CloudApplication.Domain -> D:\a\1\s\src\MES.CloudApplication.Domain\bin\Debug\net5.0\MES.CloudApplication.Domain.dll
2024-03-18T11:50:15.0053211Z ##[error]The Operation will be canceled. The next steps may not contain expected logs.
2024-03-18T11:50:24.6110630Z ##[error]The operation was canceled.
2024-03-18T11:50:24.6418805Z ##[section]Finishing: Buid solution
My build pipeline looks like this:
pool:
vmImage: 'windows-latest'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
steps:
- task: UseDotNet@2
inputs:
version: '5.0.x'
includePreviewVersions: true # Required for preview versions
displayName: Install sdk
- task: DotNetCoreCLI@2
displayName: 'dotnet restore'
inputs:
command: 'restore'
projects: '**/*.csproj'
feedsToUse: 'config'
nugetConfigPath: 'NuGet.config'
- task: DotNetCoreCLI@2
inputs:
command: 'build'
projects: '**/*.sln'
displayName: Buid solution
- task: DotNetCoreCLI@2
inputs:
command: 'test'
projects: '**/*Tests.csproj'
arguments: '--configuration $(buildConfiguration) --no-build'
displayName: Unit test
- task: DotNetCoreCLI@2
inputs:
command: 'publish'
publishWebProjects: false
projects: '**src/MES.CloudApplication.Api/*.csproj'
arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
zipAfterPublish: true
displayName: Publish artifacts
- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
displayName: Copy artifacts
All the projects are .net 5 class libraries except Api.
From the information you provided, you have 11 projects (.csproj) in the solution (.sln) need to restore packages and build on the Microsoft-Hosted agent windows-latest.
The most possible reason of this issue could be that the memory of the agent is out of storage.
Each Windows VM of Microsoft-hosted agents has a 2 core CPU, 7 GB of RAM, and 14 GB of SSD disk space. However, due to series of preinstalled software, the actually available memory would be less than that. In general, you should never rely on having more than 10GB of disk space available.
To check whether the agent is out of storage when building the projects in your solution, you can try to enable debugging logs when triggering the pipeline by one of the following ways:
When you manually trigger a new pipeline run, you can check the option "Enable system diagnostics".
Or you can set the pipeline variable "
System.Debug" to "true".variables: System.Debug: true . . .
In the output logs of pipeline run, with the debugging logs, you can see how much memory is available and how much is used during the processes.
EDIT:
From the logs you provided:
2024-03-21T05:57:44.2145857Z ##[debug]Agent environment resources - Disk: D:\ Available 11261.00 MB out of 14333.00 MB, Memory: Used 7111.00 MB out of 7167.00 MB, CPU: Usage 1.07%
We can see the memory on the agent machine is almost consumed out (Memory: Used 7111.00 MB out of 7167.00 MB).
When the memory is consumed out or not enough for the subsequent processes of build task, the task could be stuck until the jobs gets cancelled by timeout.
Since the total available memory on the Microsoft-hosted agents is not enough for building your solution, I recommended you try to set up the self-hosted agents on the local machines or VMs owned by yourself.
On your own machines, you can expand the memory and other hardware based in your demands.