Search code examples
c#asp.netaspnet-compiler

How to speed up aspnet_compiler.exe?

We have Asp.Net Mvc application for which aspnet_compiler.exe takes more than 5 minutes to run.

The Asp.Net Mvc application is question depends on ~100 smaller projects that all contribute to it different static views, Javascript files, etc... by copying them from their own project folders. These small projects are not web applications in themselves, but they contain web content in order to distribute it between the different projects.

At the end, everything is consolidated under a single web application. And then we run aspnet_compiler.exe, which takes more than 5 minutes. Ouch.

The code targets .Net Framework 4.7.2 and the web applications are not SDK style.

We obviously doing something wrong. How can we reduce this time?

EDIT 1

The whole solution takes ~14.5 minutes to build from scratch using msbuild with /m:12. The msbuild node utilization is not good. According to the detailed build summary it is:

============================== Node Utilization (IDs represent configurations) ====================================================
Timestamp:            1       2       3       4       5       6       7       8       9       10      11      12       Duration   Cumulative
...
Utilization:          22.8    11.7    17.2    7.2     52.4    16.6    12.8    13.3    13.1    29.8    34.3    14.9     Average Utilization: 20.5104451261997

It is my belief that the poor utilization is due to aspnet_compiler being called relatively late in the build at a point where all the remaining projects depend on the main web application to finish building. I have captured the project build events as chrome traces (inspired by https://github.com/rainersigwald/TraceEventLogger) here is the bird view picture: enter image description here

What looks like a long bridge in the middle is the build event for the main web application. It takes 8:55 minutes to build of which the AspNetCompiler task takes 5:15 minutes.

And so I am trying to understand why it takes so long? I am also exploring other possibilities, like arrange it as a standalone project so that msbuild could queue it on parallel with the projects depending on the main web application. But the main question remains - what can be done about the Asp.Net view precompilation to make it run faster.

Unfortunately, it is a block box for me. I do not really know anything about how it works and what are the ways to optimize it.

Solution

  • aspnet_compiler is slow and does not allow/support parallel compilation internally. It was never updated for performance, or for basic parallel compilation, and is "legacy junk" at this point. It is also not possible to run multiple instances concurrently on the same site. As aspnet_compiler generates many temp files, using a faster drive may help: however, even with a fastest SSD, the CPU is significantly under-utilized.

    Using Roslyn can speed up compilation >2x over the .NET Framework provided compiler, and is the only way I've found to improve performance of 'the core' compilation performance. It also improves site warmup speed as a result of faster compilation.

    Anecdotal performance when using Roslyn1,

    • MSBuild compiles 'two hundred assemblies' in ~5 minutes
    • aspnet_compiler compiles website in ~40m (was ~100m before Roslyn)

    I strong recommend using Rosyln, even if this doesn't make the site compilation "fast"; it also allows access to C#7 features from views and ASPX files. Ensure that Rosyln is correctly using the VBCSCompiler.exe service model.

    It is more efficient, assuming there are no compilation errors to dig through, to compile an IIS site once instead of once for each application in the site.

    For CI/CD pipelines, use release builds if possible. Having both release builds and release web.config settings (ie. debug="false") results in faster compilation. Results appear to be more mixed when using release web.config settings with debug builds. YMMV.

    Anecdotal performance when using release vs debug builds,

    • CI/CD server build for debug in ~35m
    • CI/CD server build for release in ~25m

    For local development, use in-place compilation if possible. In-place compilation allows the option for incremental compilation, where only modified files are recompiled by default.

    Anecdotal performance when using an in-place compilation for local development,

    • first compilation in-place in ~40m
    • second compilation with no files changed in ~10s

    An incremental in-place compilation can result in some errors, such as if ABI in libraries breaks; a full recompilation (-c option) will clear the issue. CI/CD builds should probably always perform a full compilation.

    Update "JScript" files to C#/VB.NET

    Neither Roslyn nor CodeDom apply for this legacy language, and appearance of such pages must run through some old compiler process which is much, much, slower.

    Unfortunately, aspnet_compiler also lacks sufficient output generation to determine which part(s) of a site are taking the most time to compiler.

    1The website I am dealing with currently has 189 compiled assemblies, 97 .INC and .ASP classic files, 1580 .ASPX files (mix of C# WebForms and 'script'), and 1221 .CSHTML files. Final .COMPILED file count in IIS is 5788. Performance improvements will likely vary based on a number of factors including types of files and dependency graphs.

    Cliff notes on setting up Roslyn for aspnet_compiler:

    • Download the Rosyln CodeDom package (Microsoft.CodeDom.Providers.DotNetCompilerPlatform). This includes both the CodeDom and a build of Roslyn.

    • Ensure the Roslyn directory in the package ends up in bin/roslyn, however such fits into the build system. Here there is a manual pre-build copy step. Likewise, the CodeDom bridge assembly must be in bin/.

    • Update the system.codedom/compilers section in the web.config file as shown in the example. If not using VB.NET, that compiler entry can be removed along with the VB-specific Roslyn analyzers. (Note that below I've set /langversion:7 for C#7 features, the highest set officially supported in any .NET Framework target.)

      <system.codedom>
    <compilers>
      <compiler language="c#;cs;csharp" extension=".cs"
        type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.CSharpCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
        warningLevel="4" compilerOptions="/langversion:7 /nowarn:1659;1699;1701"/>
      <compiler language="vb;vbs;visualbasic;vbscript" extension=".vb"
        type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.VBCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
        warningLevel="4" compilerOptions="/langversion:14 /nowarn:41008 /define:_MYTYPE=\&quot;Web\&quot; /optionInfer+"/>
    </compilers>
</system.codedom>

    Verifying that Roslyn is invoked correctly:

    Kick off a site build. During compilation there should be many csc.exe instances launched. However, these executable instances should not take up much memory. Instead, there should be another executable called VBCSCompiler.exe that runs from the bin/roslyn directory (Visual Studio may spawn it's own Roslyn compilation server as well). This executable is the compilation server that the csc.exe executables send requests to. Having the long-running compilation VBCSCompiler.exe server model is essential for the largest performance benefits of Roslyn as it avoids the relatively costly continual respawning of the compiler itself. If the compilation server cannot be started, the csc.exe processes will fallback to do the heavy compilations themselves.