Search code examples
unicodecommand-lineinputwindows-console

How can I use Unicode characters on the Windows command line?

We have a project in Team Foundation Server (TFS) that has a non-English character (š) in it. When trying to script a few build-related things, we've stumbled upon a problem; we can't pass the š letter to the command-line tools. The command prompt or what not else messes it up, and the tf.exe utility can't find the specified project.

I've tried different formats for the .bat file (ANSI, UTF-8 with and without BOM) as well as scripting it in JavaScript (which is Unicode inherently), but no luck. How do I execute a program and pass it a Unicode command line?

Solution

  • My background: I have used Unicode input/output in a console for years (and do it a lot daily. Moreover, I develop support tools for exactly this task). There are very few problems, as far as you understand the following facts/limitations:

    • CMD and “console” are unrelated factors. CMD.exe is a just one of programs which are ready to “work inside” a console (“console applications”).
    • AFAIK, CMD has perfect support for Unicode; you can enter/output all Unicode chars when any code page is active.
    • Windows’ console has a lot of support for Unicode — but it is not perfect (just “good enough”; see below).
    • chcp 65001 is very dangerous. Unless a program was specially designed to work around defects in the Windows’ API (or uses a C runtime library which has these workarounds), it would not work reliably. Windows 8 fixes ½ of these problems with cp65001, but the rest is still applicable to Windows 10.
    • I work in Windows-1252. As I already said: To input/output Unicode in a console, one does not need to set the code page.

    The details

    • To read/write Unicode to a console, an application (or its C runtime library) should be smart enough to use not the File-I/O API, but the Console-I/O API. (For an example, see how Python does it.)
    • Likewise, to read Unicode command-line arguments, an application (or its C runtime library) should be smart enough to use the corresponding API.
    • Console font rendering supports only Unicode characters in BMP (in other words: below U+10000). Only simple text rendering is supported (so European — and some East Asian — languages should work fine — as far as one uses precomposed forms). There is a [minor] fine print here for East Asian and for characters U+0000, U+0001, U+30FB.]

    Practical considerations

    • The defaults on Window are not very helpful. For best experience, one should tune up three pieces of configuration:

      • For output: a comprehensive console font. For best results, I recommend my builds. (The installation instructions are present there — and also listed in other answers on this page.)
      • For input: a capable keyboard layout. For best results, I recommend my layouts.
      • For input: allow hexadecimal input of Unicode.

    • One more gotcha with “Pasting” into a console application (very technical):

      • Hexadecimal input delivers a character on KeyUp of Alt; all the other ways to deliver a character happen on KeyDown; so many applications are not ready to see a character on KeyUp. (Only applicable to applications using the Console-I/O API.)
      • Conclusion: many applications would not react on hexadecimal input events.
      • Moreover, what happens with a “Pasted” character depends on the current keyboard layout: if the character can be typed without using prefix keys (but with arbitrary complicated combination of modifiers, as in Ctrl + Alt + AltGr + Kana + Shift + Gray) then it is delivered on an emulated key press. This is what any application expects — so pasting anything which contains only such characters is fine.
      • However, the “other” characters are delivered by emulating hexadecimal input.

      Conclusion: unless your keyboard layout supports input of a lot of characters without prefix keys, some buggy applications may skip characters when you Paste via Console’s UI: Alt + Space, E + P. (This is why I recommend using my keyboard layouts!)

    One should also keep in mind that the “alternative, ‘more capable’ consoles” for Windows are not consoles at all. They do not support Console-I/O APIs, so the programs which rely on these APIs to work would not function. (The programs which use only “File-I/O APIs to the console file handles” would work fine, though.)

    One example of such non-console is a part of Microsoft’s PowerShell. I do not use it; to experiment, press and release the Windows key, and then type powershell.

    (On the other hand, there are programs such as ConEmu or ANSICON which try to do more: they “attempt” to intercept the Console-I/O APIs to make “true console applications” work too. This definitely works for toy example programs; in real life, this may or may not solve your particular problems. Experiment.)

    Summary

    • set font, keyboard layout (and optionally, allow hexadecimal input).

    • use only programs which go through the Console-I/O APIs, and accept Unicode command-line arguments. For example, any Cygwin-compiled program should be fine. As I already said, CMD is fine too.

    Update: Initially, for a bug in cp65001, I was mixing up Kernel and CRTL layers (Update 2: and Windows user-mode API!). Also: Windows 8 fixes one half of this bug; I clarified the section about “better console” application, and added a reference to how Python does it.