TeamCity ".Net Process Runner" hangs

Question

We have started migrating our one of several projects to team city as part of CI. Below is how we have setup teamcity build. We are trying to deploy WebSite.

1) Build Step 1 (Package installation)

Using "command line " runner type install required package.

2) Build Step 2 (Build)

Using Runner type "Visual Studio (sln)" (Visual Studio 2010) build website.

3) Build Step 3 (Deploy Web Site)

Using ".Net Process Runner", deployer.exe (x86 built with .Net Framework 4) deploy site.

Deployer.exe reads config file. Config file contains "BuildId", "Environment" and "Servers" where we want build to be pushed.

<buildType id="bt52">
    <env name="Debug">
        <server path="SERVER1" />
    </env>
    <env name="QA">
    <server path="SERVER2" />
    <server path="SERVER3" />
</env>
<env name="UAT">
    <server path="SERVER4" />
    <server path="SERVER5" />
</env>
</buildType>

Deployer.exe is called with required parameters as below. Which reads config and deploys site to Server2 and Server3.

Deployer.exe "bt52" "QA" "siteQA" "E:\BuildAgent\work\2483052e33e5e1e8\src\diy\" msdeploy.exe

Problem area is step #3.

When we run deployer.exe using .Net process runner as part of team city we see its hanging and not responsind sometime even for 45 minutes. When we try to execute same deployer.exe from build server using command line script executes within couple of seconds.

E:\TeamCity_custom_applications\deployer>Deployer.exe farm1-1 QA siteQA E:\BuildAgent\work\2483052e33e5e1e8\src\diy\ msdeploy.exe

Info

: Processing batch run ... Info : Processing command ...msdeploy.exe -verb:sync -source:contentPath="E:\BuildAgent\work\2483052e33e5e1e8\src\diy\" -dest:contentPath="siteQA",wmsvc="SERVER2",userName="*****",password="******",authType="Basic"-skip:objectName=filePath,absolutePath=web.config -skip:objectName=dirPath,absolutePath="bin" -enableRule:DoNotDeleteRule -allowUntrusted Info : output >>Total changes: 0 (0 added, 0 deleted, 0 updated, 0 parameters changed, 0 bytes copied) Info : error >>(none) Info : ExitCode >> 0 Info :

Processing command ...msdeploy.exe -verb:sync -source:contentPath="E:\BuildAgent\work\2483052e33e5e1e8\src\diy\" -dest:contentPath="siteQA",wmsvc="SERVER3",userName="******",password="******",authType="Basic" -skip:objectName=filePath,absolutePath=web.config -skip:objectName=dirPath,absolutePath="bin" -enableRule:DoNotDeleteRule -allowUntrusted Info : output >>Total changes: 0 (0 added, 0 deleted, 0 updated, 0 parameters ch anged, 0 bytes copied) Info : error >>(none) Info : ExitCode >> 0

Info: Deploy Script Complete.

One more thing we observed is running deployer.exe through teamcity I see that site content gets copied but only for 1 server and teamcity build status stays in "Running" mode. I am wondering if someone can please put little bit of insight on how can I look into this issue.

Update 1:

Thanks for your time looking into it !! What we ended up doing is, Instead of running command "msdeploy.exe" from "cmd.exe" we added "msdeploy.exe" location as Environment variable and executed "msdeploy.exe" in loop for # of servers. This has resolved issue of hanging. Now I am just curious to know why would it behave in such manner where if you execute "msdeploy.exe" from "cmd.exe" it would hang while running directly "msdeploy.exe" it would execute successfully. Any insight into same would be greatly appreciated.

Update 2:

I have added image which explains behavior using process explorer. If we kill msdeploy.exe from process explorer than for next all deployments to that server will not have the issue of build hanging. Please see below image

Answer 1

Work around we did is, we observed msdeploy doesn't take more than 3-5 seconds to execute and deploy (Even for our biggest project which is almost 300mb website). So we set timeout of 20 seconds. So far since last 1 weeks we have not seen any issue with it and hopefully it will not cause more trouble but still we are not sure why such behavior.