SetConsoleMode returning false when enabling ANSI color under Windows 10

Question

I've recently upgraded one of my PCs from windows 7 pro to windows 10 pro. I'm trying to run some code that I know is working on another windows 10 machine (that was win10 pro from the beginning). In both cases, I'm using Visual Studio Community 2017, target platform is Windows 10 and the platform toolset is v141. I've tried a few different windows SDK versions, which hasn't changed the results. Debug and release mode builds behave the same way. The application is run from visual studio with the usual "start without debugging". Running it from a console externally to visual studio makes no difference.

This is pretty much the example code that MS provide for enabling ANSI colour mode in a terminal application:

#include <Windows.h>
#include <iostream>

int main()
{
  HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE);
  DWORD consoleMode;
  GetConsoleMode(hConsole, &consoleMode);
  consoleMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;

  if (!SetConsoleMode(hConsole, consoleMode))
  {
    auto e = GetLastError();
    std::cout << "error " << e << "\n";
  }
  else
    std::cout << "\x1b[32mgreen\n";
}

On the upgraded machine, SetConsoleMode returns false, and GetLastError returns 87 ("The parameter is incorrect") and ANSI colour codes are not handled by the console in subsequent output. On the other machine, this all succeeds just fine and ansi colors are rendered as expected.

Opening up a console window on the upgraded machine shows Version 10.0.10240, and on the other machine it shows Version 10.0.14393. Although I'm slightly suprised that the upgraded machine isn't running a more recent version, the version it is running should support ANSI color.

What could explain the difference in behavior between the two machines?

Answer 1

Support for ANSI escape sequences in console windows was not added until Windows 10 Threshold 2 (TH2), also known as the "November Update", which is version 1511 and has the build number 10586.

Your "other machine", with version 10.0.14393, therefore has full support when you pass the appropriate flag to SetConsoleMode. However, the "upgraded machine", with version 10.0.10240, does not recognize the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag and is therefore returning an "Invalid Parameter" error.

In fact, this is the officially recommended way to check whether the OS supports the extended console features:

The following code provides an example of the recommended way to enable virtual terminal processing for an application. The intent of the sample is to demonstrate:

1. The existing mode should always be retrieved via GetConsoleMode and analyzed before being set with SetConsoleMode.
2. SetConsoleMode returning with STATUS_INVALID_PARAMETER is the current mechanism to determine when running on a down-level system. An application receiving STATUS_INVALID_PARAMETER with one of the newer console mode flags in the bit field should gracefully degrade behavior and try again.

As a workaround on downlevel systems, you could use ANSICon or ConEmu.