nasm can generate only machine code by -f bin
option. I wanted to do the same thing on masm but I can't find how.
It looks always the object code generated is in COFF format and doesn't seem to be such option to turn it off.
Any advice?
You can't. But if you goal is simply to write a block of binary to a file, you can easily abuse an assembler into helping you.
Write your assembler program with two parts
The part you want to produce as a binary block. Give it a starting label, BLOCK, and and ending label, BLOCKEND.
A part which gains control when the executable is loaded, and makes a call on the OS (e.g, Windows) to write the block to a file; when done, it simply exits. The actual calls to do this aren't actually very difficult to write; you'll likely have 50-100 lines of code in this section.