Generically adding object file to Link task in MSBuild in addition to existing linked files

Question

TL;DR

How do I feed an additional object file created outside of the normal build process back into the (MSBuild) build process in addition to whatever else is going on?

/TL;DR

The reason why is best explained with some background information (WARNING: wall of text looming).

We develop applications based on the Qt5 Framework. A part of the process is to embed translation (.qm) files into a resource dll using the Qt resource manager. This has worked great for many years, but recently we bumped into the (apparently wellknown and dreaded) "C1060: Compiler is out of heap space" error.

This happens because Qt's resource compiler (RCC) creates an increasingly large .cpp file containing the embedded resources. At some point that file is too large for the compiler to handle (we hit the limit at approx. 60 MB).

Googling the issue usually results in recommendations to use "CONFIG += resources_big" (undocumented by the way) in the project's .pro file. The problem for us is, we do not use .pro files, but Visual Studio's MSBuild.

Some further googling shows what "CONFIG += resources_big" actually does.

Instead of generating a (huge) .cpp file using the usual syntax

rcc.exe my_resources.qrc -o qrc_my_resources.cpp

a three-stage approach is applied using the (equally undocumented) RCC parameters -pass 1 and -pass 2.

First you create a special .cpp file which allocates the necessary space

rcc.exe my_resources.qrc -pass 1 -o qrc_my_resources.cpp

Then you explicitly call the compiler on this particular .cpp file

cl.exe /c /Zl qrc_my_resources.cpp -Foqrc_my_resources.tmp

This creates an object file padded mainly with zeros. Parameter /c causes that no linking occurs and /Zl makes sure no default library calls are embedded (otherwise you'd get linker errors later).

The third and final stage is to call RCC again and have it fill the empty space in the object file with the actual binary data.

rcc.exe my_resources.qrc -pass 2 -temp qrc_my_resources.tmp -o qrc_my_resources.obj

Now we have our final object file containing all the resources without causing the dreaded C1060 error. However, we now also have a problem because we need to tell the linker to use our object file in addition to the automatically generated list of object files provided by cl.exe for all the files that weren't compiled with /c.

Further complications arise, since an ex-employee created a very intricate MSBuild solution. Specifically to deal with automatically RCC'ing Qt resource files he created the following properties (the TemporaryFileName and ObjectFileName elements are my addition).

<PropertyGroup>
  <QtToolsIntermediateDir>$(IntermediateOutputPath)GeneratedFiles</QtToolsIntermediateDir>
</PropertyGroup>
<ItemDefinitionGroup>
  <QtRCC>
    <OutputFileName>$(QtToolsIntermediateDir)\qrc_%(FileName).cpp</OutputFileName>
    <TemporaryFileName>$(IntermediateOutputPath)\qrc_%(FileName).tmp</TemporaryFileName>
    <ObjectFileName>$(IntermediateOutputPath)\qrc_%(FileName).obj</ObjectFileName>
  </QtRCC>
</ItemDefinitionGroup>

In a special .targets file he defined the automatic resource compilation like this

<Target Name="QtRCC"
        BeforeTargets="CustomBuild"
        DependsOnTargets="QtToolsEnsureOutputDir"
        Inputs="%(QtRCC.Dependencies)"
        Outputs="%(QtRCC.OutputFileName)">
  <Message Text="RCC'ing @(QtRCC)..."/>
  <Exec Command="$(MSBuildThisFileDirectory)tools\rcc.exe -name %22%(QtRCC.FileName)%22 -no-compress %22%(FullPath)%22 -o %(QtRCC.OutputFileName)"/>
</Target>

which results in the "normal" compilation process (the one resulting in a huge .cpp file).

I exchanged that for the three-stage approach described above

  <Exec Command="$(MSBuildThisFileDirectory)tools\rcc.exe -name %22%(QtRCC.FileName)%22 -no-compress %22%(FullPath)%22 -pass 1 -o %(QtRCC.OutputFileName)"/>
  <Exec Command="cl.exe -c -Zl %(QtRCC.OutputFileName) -Fo%(QtRCC.TemporaryFileName)"/>
  <Exec Command="$(MSBuildThisFileDirectory)tools\rcc.exe -name %22%(QtRCC.FileName)%22 -no-compress %22%(FullPath)%22 -pass 2 -temp %(QtRCC.TemporaryFileName) -o %(QtRCC.ObjectFileName)"/>

This works exactly as intended but again has the same problem also stated above. Namely, that I now have an object file which I need the linker to link in addition to all the automatically generated link targets.

I have found one solution, but it's one I really want to avoid. This solution is to manually add the object file containing the resources in the AdditionalDependencies section of the Link task of every project file that may use a Qt resource file.

<Link>
  <AdditionalDependencies>$(IntermediateOutputPath)\qrc_my_resources.obj;Qt5Core.lib;Qt5Gui.lib;Qt5Widgets.lib;Qt5PrintSupport.lib;%(AdditionalDependencies)</AdditionalDependencies>
</Link>

I would very much like to have this in a generic place like the aforementioned .targets or .props files. Is that possible? And if so, how?

Answer 1

I know the problem you are referring to and the easiest solution for me was to use the Qt's property sheet that are provided in QT's MS Visual Studio AddOn.

Basically, everyone will tell you to add CONFIG += resources_big" in the project's .pro file, but you do not use .pro files, but Visual Studio's MSBuild.

All the options availables in CMake are available in MSBuild but it is not documented.

Install the Qt Visual Studio Extension, once this is done, go in %APPDATA%/Local/QtMsBuild and copy all the property sheets in your project externals directory. Use those property sheets in your project Like that:

 <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Retail|x64'">
   <QtRcc>
      <ExecutionDescription>Rcc'ing %(Identity)...</ExecutionDescription>
      <OutputFile>$(ProjectDir)\GeneratedFiles\qrc_%(Filename).cpp</OutputFile>
      <!-- Enable verbose mode '-verbose' -->
      <Verbose>true</Verbose>
 <!-- Pass Number for big resources '-pass' will make obj smaller each time-->
      <PassNumber>10</PassNumber>
      <!-- Disable all compression '-no-compress' (false by default, should be false in final product) -->
      <NoCompression>false</NoCompression>
      <!-- Output a binary file for use as a dynamic source '--binary' -->
      <BinaryOutput>false</BinaryOutput>
      <!-- Run work in parallel processes or not, this was buggy last time I tried. No big time improvement, keep it off -->
      <ParallelProcess>false</ParallelProcess>
      <!-- Compress input files by <level> '-compress <level>'  
        -- The level is a int from 1 to 9 and represent the zlib compression levels 
          #define Z_NO_COMPRESSION         0
          #define Z_BEST_SPEED             1
          #define Z_BEST_COMPRESSION       9
          #define Z_DEFAULT_COMPRESSION  (-1)
        -->
      <Compression>level3</Compression>
      <!--  Threshold to consider compressing files '-threshold <level>' 
            Specifies a threshold (in bytes) to use when compressing files. 
            If the file is smaller than the threshold, it will not be compressed, independent of what the compression level is.
            Keep this unset, it is not used and be default behavior
      <CompressThreshold>3</CompressThreshold>
      -->
      <InitFuncName>QmlInitialization</InitFuncName>
    </QtRcc>

I suggest creating a small example project and adding Qt option from the UI and look at the syntax in the generated vcxproj. You will create your documentation for your own Qt project.

Make sense ?

One last thing: using the solution I provided, you can easily compile the QML for caching, by using the QmlCacheGenerate option:

Sample .vcxproj

<ItemGroup Condition="'$(Configuration)|$(Platform)'=='Retail|x64'">
    <QtRcc Include="$(QtResourceFile)" >
      <QmlCacheGenerate>true</QmlCacheGenerate>
    </QtRcc>
</ItemGroup>

BTW the -pass argument is not the ID of the pass but the number of pass to do when compiling resources. If you end up with a 10Mb cpp, do 10 pass to compile 10Mb files

Cheers.

-- Guillaume Plante Senior Programmer