Search code examples
c++windowspowershellwinapi

How to distinguish if console program is opened in Powershell or in Windows Terminal?


I'm programming a library which will make setting colors, modes, etc. easier in console program. But I've encountered a problem with Windows Terminal. For example I have a function:

void WindowsCLI::setUnderlinedFont()
{
    auto consoleHandle = GetStdHandle(STD_OUTPUT_HANDLE);
    config.underlined = true;
    SetConsoleTextAttribute(consoleHandle, getTextAttribute(config));
}

and it uses COMMON_LVB_UNDERSCORE attribute from windows.h to make text underlined. And the result of this function in powershell looks like this: enter image description here, and in Windows Terminal like this:enter image description here, so apparently in the second case my function didn't work properly. I thought that the problem is that the Windows Terminal runs in virtual terminal mode. So I made another function for virtual terminals:

void WindowsVirtualCLI::setUnderlinedFont()
{
    printf("\x1b[4m");
}

and now it didn't work for Powershell: enter image description here, and worked properly for Windows Terminal: enter image description here. But now I have another problem. How to distinguish that the program is run in Powershell or Windows Terminal. I tried using this function:

CLI& cli()
{
    DWORD consoleMode;
    auto consoleHandle = GetStdHandle(STD_INPUT_HANDLE);
    GetConsoleMode(consoleHandle, &consoleMode);

    if ((consoleMode & ENABLE_PROCESSED_OUTPUT) && (consoleMode & ENABLE_VIRTUAL_TERMINAL_PROCESSING))
    {
        return windows::WindowsVirtualCLI::getInstance();
    }
    else
    {
        return windows::WindowsCLI::getInstance();
    }
}

But it turned out that both Powershell and Windows Terminal have ENABLE_PROCESSED_OUTPUT and ENABLE_VIRTUAL_TERMINAL_PROCESSING enabled. And now, I have no other idea how can I distinguish these terminals in runtime. Do you have any idea how?

P.S. I have changed cli() method to this:

CLI& cli()
{
    DWORD consoleMode;
    auto consoleHandle = GetStdHandle(STD_OUTPUT_HANDLE);
    GetConsoleMode(consoleHandle, &consoleMode);
    SetConsoleMode(consoleHandle, consoleMode);

    if ((consoleMode & ENABLE_PROCESSED_OUTPUT) && (consoleMode & ENABLE_VIRTUAL_TERMINAL_PROCESSING))
    {
        return windows::WindowsVirtualCLI::getInstance();
    }
    else
    {
        return windows::WindowsCLI::getInstance();
    }
}

And still it doesn't work as I need to.


Solution

  • Simple, but not foolproof solution:

    Windows Terminal defines two application-specific environment variables, WT_SESSION and WT_PROFILE_ID, so you can test whether one of these variables is defined (with a non-empty value).

    According to this answer, getenv("WT_SESSION") should work in C++ for retrieving the value of that variable, if defined.

    In PowerShell itself, [bool] $env:WT_SESSION returns $true if the variable has a non-empty value.

    This approach isn't foolproof, because if you launch a regular console window (conhost.exe) from a shell running in Windows Terminal - e.g. with Start-Process cmd.exe - it'll inherit the WT_SESSION variable and thus produce a false positive in such sessions.


    More elaborate, but robust solution:

    A robust solution is to walk the chain of parent processes starting with the current process until the first process with an associated main window handle is found:

    • If that process' name is WindowsTerminal, it is safe to assume that the process is running in Windows Terminal - possibly indirectly, in a (by definition console-subsystem) process launched from the shell running directly in Windows Terminal.

    • Otherwise, it is safe to assume that a different application is hosting the process, possibly the process itself (a process running in a regular console window itself owns that window, and has a conhost.exe child process).

    The following are PowerShell implementations:

    • PowerShell (Core) 7+ implementation:
    $runningInWindowsTerminal = 
      if ($IsWindows) {
        $ps = Get-Process -Id $PID
        while ($ps -and 0 -eq [int] $ps.MainWindowHandle) { $ps = $ps.Parent }
        $ps.ProcessName -eq 'WindowsTerminal'
      } else {
        $false
      }
    
    • Windows PowerShell and cross-edition implementation:

    Unfortunately, Windows PowerShell doesn't decorate the System.Diagnostics.Process objects output by Get-Process with a .Parent property reporting the parent process, which complicates the solution and requires a Get-CimInstance call to retrieve the parent information.

    $runningInWindowsTerminal = 
      if ($env:OS -eq 'Windows_NT') {
        $ps = Get-Process -Id $PID
        while ($ps -and 0 -eq [int] $ps.MainWindowHandle) { 
          $ps = Get-Process -ErrorAction Ignore -Id (Get-CimInstance Win32_Process -Filter "ProcessID = $($ps.Id)").ParentProcessId 
        }
        $ps.ProcessName -eq 'WindowsTerminal'
      } else {
        $false
      }