How to make sure all components are deselected when using /COMPONENTSONLYONCUSTOM?

I have the following NSIS installation types:

InstType "X (recommended)"
InstType "/CUSTOMSTRING=Y (advanced mode)"
InstType /COMPONENTSONLYONCUSTOM

The idea is that installation "X" should install all components silently, whereas installation "Y" should only install components that have been selected. The components of installation "Y" should all be deselected by default. This is something I can't manage to achieve.

I have tried a lot of different scenarios to get all components deselected for installation "Y", but for some reason NSIS will always take the selection of "X" as the default for "Y". Since all components have been selected for "X", installation "Y" will have all components selected by default.

How can I make sure all components installation "Y" are deselected by default for this scenario?

Solution

The /CUSTOMSTRING InstType is special so you are maybe bending the rules a bit here, the point of custom is to let the user cherry pick sections and end up with something different than any of your predefined InstTypes. It does not really have a default, it is based on the previous InstType the user selected (In your case it is always X).

!include LogicLib.nsh
!include Sections.nsh
!include WinMessages.nsh

Page Components
Page InstFiles

!define ITSIN_X 1 ; SectionIn ID's are 1 based
InstType "X (recommended)"
InstType "/CUSTOMSTRING=Y (advanced mode)" ; The "special" custom InstType
InstType /COMPONENTSONLYONCUSTOM

Section "A" SID_A
SectionIn ${ITSIN_X}
DetailPrint a
SectionEnd

Section "B" SID_B
SectionIn ${ITSIN_X}
DetailPrint b
SectionEnd

Function .onSelChange
/*
UNDOCUMENTED HACK!
We are going to check if the current InstType is the custom type even if the current section "selection" matches another InstType (GetCurInstType returns non-custom if possible)
*/
FindWindow $9 "#32770" "" $HWNDPARENT
FindWindow $9 "ComboBox" "" $9
SendMessage $9 ${CB_GETCURSEL} 0 0 $0
SendMessage $9 ${CB_GETITEMDATA} $0 0 $0
${If} $0 = ${NSIS_MAX_INST_TYPES} ; The custom InstType?
${AndIf} $1 <> $0 ; Only do the unselect hack on InstType changes (BUGBUG: Should really set $1 to something in the page create/show callback)
!if 1 ; If you only have a few sections you can just use their ID
    !insertmacro UnselectSection ${SID_A}
    !insertmacro UnselectSection ${SID_B}
!else ; ...or use a loop if you are lazy
    StrCpy $2 0
    ClearErrors
    loop:
        SectionGetFlags $2 $3
        IfErrors end
        !insertmacro UnselectSection $2 ; You could check SectionGetText if you need to skip hidden sections here
        IntOp $2 $2 + 1
        Goto loop
    end:
!endif
${EndIf}
StrCpy $1 $0 ; Save the current InstType so we can tell if it changes
FunctionEnd