Is there any way to make MASM generate COMDATs for functions, so that unused functions are removed by the linker?
(i.e. I'm looking for the equivalents of /Gy for MASM.)
Not straightforward, but doable; discussed here and here.
The first step involves putting each function into a separate segment with names like .text$a, .text$b, etc. This way, the assembler won't unite them into a single .text section, but the linker eventually will; there's a special rule in Microsoft linkers regarding the stuff past the $ character in the section name. The assembler will emit an .obj file with multiple code sections. I've tried that, I can confirm that it does. At least one flavor of MASM does. :)
Then they suggest running an utility over an object file that will mark your sections as COMDATs. The said utility seems to be lost to time and bit decay, but its action can be roughly deduced. It reads and parses a COFF .obj file, goes through sections and slaps a COMDAT flag on all .text sections. I assume it's just a flag; could be more. As a first step to its recreation, I'd suggest compiling a C file with /Gy then without, and comparing the two .obj files via some low-level PE/COFF browser. I didn't go this far, since my scenario was rather different.