In addition to the bog-standard LZNT1 compression on NTFS, Windows 10 has a new form of compression called system compression. This mode of compression is "optimized" for files that are read-only, namely executable files. It is used for the Compact OS feature, although one can also manually invoke it through compact.exe /EXE.
But what is the programmatic way to get and set the new type of compression status? I tried the old FSCTL_GET_COMPRESSION on the file, but it always gives 0 for COMPRESSION_FORMAT_NONE.
Okay, here is what I have right now, after throwing compact.exe at Ghidra.
Besides the FSCTL_GET_COMPRESSION / FSCTL_SET_COMPRESSION pair of DeviceIoControl, compact also calls FSCTL_GET_EXTERNAL_BACKING and FSCTL_SET_EXTERNAL_BACKING. The documentation explains the compression information is stored in the FILE_PROVIDER_EXTERNAL_INFO_V1 structure for the input/output. The fields are pretty well documented.
I am a bit too lazy to add a proper struct to Ghidra to verify this finding right now, but I am pretty sure this is the way.
A bit unorthodox is the way compact.exe calls the FSCTLs. Although the functions are documented to only use the NT APIs, the Win32 DeviceIoControl is used here and works just fine. The error codes are a bit strange, including undocumented stuff like 0x156 and 0x158. I need to find some time to throw stuff at RtlNtStatusToDosError before I dare to actually use these with Win32 handles.
FWIW, the set call is found at FUN_1400030e0 (0x158 and 0x13d are acceptable?), the get calls are at FUN_140002074 (0xea?) and FUN_1400040d, and a delete call is at FUN_1400040dc. This is for version 10.0.19041.1.