Search code examples
c++assemblyvisual-c++compiler-errorsbootloader

CL.EXE (v8.00c) produces an "Out of memory" error when compiling files


I'm building the bootloader from this CodeProject article. The downloadable file contains source code for a bootloader and a patched version of CL.EXE, ML.EXE and LINK.EXE.

When CL.EXE is run it displays:

Microsoft (R) C/C++ Optimizing Compiler Version 8.00c

Copyright (c) Microsoft Corp 1984-1993. All rights reserved.

bootmain.cpp

Out of memory

The %errorlevel% return value 2. The file is not compiled to an object file when the Out of memory error occurs.

Note: I'm using version 8.00c of CL.EXE that was part of Visual C++ 1.52c which was the last version of the Visual C++ compiler that could produce 16-bit executables and binaries.


If I reduce BootMain.cpp to this minimal file:

extern "C" void BootMain()
{
    return;
}

I still get an Out of Memory error when I compile it to an object file with:

.\VC152\CL.EXE /AT /G2 /Gs /Gx /c /Zl BootMain.cpp

Solution

  • Some antiquated tools can't handle long environment variables like PATH or INCLUDE which can become unwieldly in modern Windows development environments. Your CL.EXE happens to be a Win32 PE executable so should run on even 64-bit Windows as a 32-bit application. Since CL.EXE worked for me in DOSBox I had to assume there was some external factor for the Out of Memory error when run directly on Windows. My first thought was the environment variables so I reduced mine to the bare minimum.

    As an experiment one could temporarily put these commands in BUILD.BAT at the beginning to see if environment variables are a problem:

    set PATH=
    set INCLUDE=
    

    If this solves your problem you may wish to put these lines at the top of your BUILD.BAT to save the old environment variables and use empty ones:

    @set OLDINC=%INCLUDE%
    @set OLDPATH=%PATH%
    @set PATH=
    @set INCLUDE=
    

    To restore these environment variables place these lines at the end:

    @set INCLUDE=%OLDINC%
    @set PATH=%OLDPATH%
    @set OLDINC=
    @set OLDPATH=
    

    Miscellaneous

    • If you want LINK.EXE to stop prompting you unnecessarily for information place a ; (semicolon) at the end of the linker command line to force it to use default values. It would look like:

      .\VC152\LINK.EXE /T /NOD StartPoint.obj bootmain.obj cdisplay.obj cstring.obj;
      
    • An @ at the beginning of a batch file line suppresses the command being echoed to standard output (ie. console)