How to check if an handle should be closed?

In case which ShellExecuteEx returns false, should the handle be closed?:

function EditAndWait(const AFileName : string) : boolean;
var
  Info: TShellExecuteInfo;
begin
  FillChar(Info, SizeOf(Info), 0);
  Info.cbSize := SizeOf(Info);
  Info.lpVerb := 'edit';
  Info.lpFile := PAnsiChar(AFileName);
  Info.nShow := SW_SHOW;
  Info.fMask := SEE_MASK_NOCLOSEPROCESS;
  Result := ShellExecuteEx(@Info);
  if(Result) then 
  begin
    WaitForSingleObject(Info.hProcess, Infinite);
    CloseHandle(Info.hProcess);
  end else
  begin
     //should I close the process handle?
  end;
end;

More generally, how can I check if an handle should be closed?

Solution

You are only returned a process handle if:

You included SEE_MASK_NOCLOSEPROCESS, and
The function call succeeded, and
The action was resolved by creating a new process.

In case the first two conditions hold, but not the third, then you will be handled back a process handle with value zero. So your code should be:

Result := ShellExecuteEx(@Info);
if Result and (Info.hProcess<>0) then 
begin
  WaitForSingleObject(Info.hProcess, Infinite);
  CloseHandle(Info.hProcess);
end;

If we were being very pedantic we might look for error checking on WaitForSingleObject and CloseHandle. Frankly though, I find it hard to get excited about that in this instance. What are the possible failure modes that could be recovered from?

You might well ask what I mean by:

The action was resolved by creating a new process.

Well, it's entirely possible for a shell action to be resolved by re-cycling an existing process. In which case you may not be returned a process handle. And that puts the kibosh on your code because you have nothing to wait on, never mind no handle to close. You'll just have to accept that such scenarios are beyond you.

The documentation has this to say:

SEE_MASK_NOCLOSEPROCESS

Use to indicate that the hProcess member receives the process handle. This handle is typically used to allow an application to find out when a process created with ShellExecuteEx terminates. In some cases, such as when execution is satisfied through a DDE conversation, no handle will be returned. The calling application is responsible for closing the handle when it is no longer needed.

Finally, may I congratulate you on taking seriously the issues of error checking and leak avoidance. So many developers seem to ignore this issue, no matter how many times they are told. It's nice that you have listened to comments at recent questions and made the effort to improve your code. Well done!