HOME

TheInfoList



OR:

Microsoft Microsoft Corporation is an American multinational corporation and technology company, technology conglomerate headquartered in Redmond, Washington. Founded in 1975, the company became influential in the History of personal computers#The ear ...
was one of the first companies to implement
Unicode Unicode or ''The Unicode Standard'' or TUS is a character encoding standard maintained by the Unicode Consortium designed to support the use of text in all of the world's writing systems that can be digitized. Version 16.0 defines 154,998 Char ...
in their products.
Windows NT Windows NT is a Proprietary software, proprietary Graphical user interface, graphical operating system produced by Microsoft as part of its Windows product line, the first version of which, Windows NT 3.1, was released on July 27, 1993. Original ...
was the first operating system that used "wide characters" in
system call In computing, a system call (syscall) is the programmatic way in which a computer program requests a service from the operating system on which it is executed. This may include hardware-related services (for example, accessing a hard disk drive ...
s. Using the (now obsolete) UCS-2 encoding scheme at first, it was upgraded to the
variable-width encoding A variable-width encoding is a type of character encoding scheme in which codes of differing lengths are used to encode a character set (a repertoire of symbols) for representation, usually in a computer. Most common variable-width encodings are ...
UTF-16 UTF-16 (16-bit Unicode Transformation Format) is a character encoding that supports all 1,112,064 valid code points of Unicode. The encoding is variable-length as code points are encoded with one or two ''code units''. UTF-16 arose from an earli ...
starting with
Windows 2000 Windows 2000 is a major release of the Windows NT operating system developed by Microsoft, targeting the server and business markets. It is the direct successor to Windows NT 4.0, and was Software release life cycle#Release to manufacturing (RT ...
, allowing a representation of additional planes with surrogate pairs. However Microsoft did not support
UTF-8 UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode Transformation Format 8-bit''. Almost every webpage is transmitted as UTF-8. UTF-8 supports all 1,112,0 ...
in its API until May 2019. Before 2019, Microsoft emphasized UTF-16 (i.e. -W API), but has since recommended to use
UTF-8 UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode Transformation Format 8-bit''. Almost every webpage is transmitted as UTF-8. UTF-8 supports all 1,112,0 ...
(at least in some cases), on Windows and
Xbox Xbox is a video gaming brand that consists of four main home video game console lines, as well as application software, applications (games), the streaming media, streaming service Xbox Cloud Gaming, and online services such as the Xbox networ ...
(and in other of its products), even states "UTF-8 is the universal code page for internationalization ndUTF-16 .. isa unique burden that Windows places on code that targets multiple platforms. .Windows smoving forward to support UTF-8 to remove this unique burden esultingin fewer internationalization issues in apps and games". A large amount of Microsoft documentation uses the word "Unicode" to refer explicitly to the UTF-16 encoding. Anything else, including UTF-8, is not "Unicode" in Microsoft's outdated language (while UTF-8 and UTF-16 are both Unicode according to
the Unicode Standard Unicode or ''The Unicode Standard'' or TUS is a character encoding standard maintained by the Unicode Consortium designed to support the use of text in all of the world's writing systems that can be digitized. Version 16.0 defines 154,998 ch ...
, or encodings/"transformation formats" thereof).


In various Windows families


Windows NT based systems

Current Windows versions and all back to
Windows XP Windows XP is a major release of Microsoft's Windows NT operating system. It was released to manufacturing on August 24, 2001, and later to retail on October 25, 2001. It is a direct successor to Windows 2000 for high-end and business users a ...
and prior
Windows NT Windows NT is a Proprietary software, proprietary Graphical user interface, graphical operating system produced by Microsoft as part of its Windows product line, the first version of which, Windows NT 3.1, was released on July 27, 1993. Original ...
(3.x, 4.0) are shipped with
system libraries In computing, a library is a collection of resources that can be leveraged during software development to implement a computer program. Commonly, a library consists of executable code such as compiled functions and classes, or a library can ...
that support string
encoding In communications and Data processing, information processing, code is a system of rules to convert information—such as a letter (alphabet), letter, word, sound, image, or gesture—into another form, sometimes data compression, shortened or ...
of two types: 16-bit "Unicode" (
UTF-16 UTF-16 (16-bit Unicode Transformation Format) is a character encoding that supports all 1,112,064 valid code points of Unicode. The encoding is variable-length as code points are encoded with one or two ''code units''. UTF-16 arose from an earli ...
since
Windows 2000 Windows 2000 is a major release of the Windows NT operating system developed by Microsoft, targeting the server and business markets. It is the direct successor to Windows NT 4.0, and was Software release life cycle#Release to manufacturing (RT ...
) and a (sometimes multibyte) encoding called the "
code page In computing, a code page is a character encoding and as such it is a specific association of a set of printable character (computing), characters and control characters with unique numbers. Typically each number represents the binary value in a s ...
" (or incorrectly referred to as ''
ANSI The American National Standards Institute (ANSI ) is a private nonprofit organization that oversees the development of voluntary consensus standards for products, services, processes, systems, and personnel in the United States. The organiz ...
code page''). 16-bit functions have names suffixed with 'W' (from "wide") such as SetWindowTextW. Code page oriented functions use the suffix 'A' for "ANSI" such as SetWindowTextA (some other conventions were used for APIs that were copied from other systems, such as _wfopen/fopen or wcslen/strlen). This split was necessary because many languages, including C, did not provide a clean way to pass both 8-bit and 16-bit strings to the same function.
Microsoft Microsoft Corporation is an American multinational corporation and technology company, technology conglomerate headquartered in Redmond, Washington. Founded in 1975, the company became influential in the History of personal computers#The ear ...
attempted to support Unicode "portably" by providing a "UNICODE" switch to the compiler, that switches unsuffixed "generic" calls from the 'A' to the 'W' interface and converts all string constants to "wide" UTF-16 versions. This does not actually work because it does not translate UTF-8 outside of string constants, resulting in code that attempts to open files just not compiling. Earlier, and independent of the "UNICODE" switch, Windows also provided the Multibyte Character Sets (MBCS) API switch. This changes some functions that don't work in MBCS such as strrev to an MBCS-aware one such as _mbsrev.


Windows CE

In (the now discontinued)
Windows CE Windows CE, later known as Windows Embedded CE and Windows Embedded Compact, is a discontinued operating system developed by Microsoft for mobile and embedded devices. It was part of the Windows Embedded family and served as the software foun ...
, UTF-16 was used almost exclusively, with the 'A' API mostly missing. A limited set of ANSI API is available in Windows CE 5.0, for use on a reduced set of locales that may be selectively built onto the runtime image.


Windows 9x

In 2001, Microsoft released a special supplement to Microsoft's old
Windows 9x Windows 9x is a generic term referring to a line of discontinued Microsoft Windows operating systems released from 1995 to 2000 and supported until 2006, which were based on the kernel introduced in Windows 95 and modified in succeeding version ...
systems. It includes a dynamic link library, 'unicows.dll', (only 240 KB) containing the 16-bit flavor (the ones with the letter W on the end) of all the basic functions of Windows API. It is merely a translation layer: SetWindowTextW will simply convert its input using the current codepage and call SetWindowTextA.


UTF-8

Microsoft Windows (
Windows XP Windows XP is a major release of Microsoft's Windows NT operating system. It was released to manufacturing on August 24, 2001, and later to retail on October 25, 2001. It is a direct successor to Windows 2000 for high-end and business users a ...
and later) has a code page designated for
UTF-8 UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode Transformation Format 8-bit''. Almost every webpage is transmitted as UTF-8. UTF-8 supports all 1,112,0 ...
, code page 65001 or CP_UTF8. For a long time, it was impossible to set the locale code page to 65001, leaving this code page only available for a) explicit conversion functions such as MultiByteToWideChar and/or b) the Win32 console command chcp 65001 to translate stdin/out between UTF-8 and UTF-16. This meant that "narrow" functions, in particular fopen (which opens files), couldn't be called with UTF-8 strings, and in fact there was no way to open all possible files using fopen no matter what the locale was set to and/or what bytes were put in the string, as none of the available locales could produce all possible UTF-16 characters. This problem also applied to all other APIs that take or return 8-bit strings, including Windows ones such as SetWindowText. Programs that wanted to use UTF-8, in particular code intended to be portable to other operating systems, needed a workaround for this deficiency. The usual work-around was to add new functions to open files that convert UTF-8 to UTF-16 using MultiByteToWideChar and call the "wide" function instead of fopen. Dozens of multi-platform libraries added wrapper functions to do this conversion on Windows (and pass UTF-8 through unchanged on others), an example is a proposed addition to Boost, . Another popular work-around was to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library. None of these workarounds are considered good, as they require changes to the code that works on non-Windows. In April 2018 (or possibly November 2017), with insider build 17035 (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8. This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings. However this is a system-wide setting and a program cannot assume it is set. In May 2019, Microsoft added the ability for a program to set the code page to UTF-8 itself, allowing programs written to use UTF-8 to be run by non-expert users. , Microsoft recommends programmers use UTF-8 (e.g. instead of any other 8-bit encoding), on Windows and
Xbox Xbox is a video gaming brand that consists of four main home video game console lines, as well as application software, applications (games), the streaming media, streaming service Xbox Cloud Gaming, and online services such as the Xbox networ ...
, and may be recommending its use instead of UTF-16, even stating "UTF-8 is the universal code page for internationalization ndUTF-16 .is a unique burden that Windows places on code that targets multiple platforms." Microsoft does appear to be transitioning to UTF-8, stating it previously emphasized its alternative, and in
Windows 11 Windows 11 is a version of Microsoft's Windows NT operating system, released on October 5, 2021, as the successor to Windows 10 (2015). It is available as a free upgrade for devices running Windows 10 that meet the #System requirements, Windo ...
some system files are required to use UTF-8 and do not require a Byte Order Mark. Notepad can now recognize UTF-8 without the Byte Order Mark, and can be told to write UTF-8 without a Byte Order Mark. Some other Microsoft products are using UTF-8 internally, including Visual Studio and their
SQL Server 2019 The history of Microsoft SQL Server begins with the first Microsoft SQL Server database product – SQL Server v1.0, a 16-bit relational database for the OS/2 operating system, released in 1989. Versions Detailed history Genesis By the late ...
, with Microsoft claiming 35% speed increase from use of UTF-8, and "nearly 50% reduction in storage requirements."


String constants in Visual Studio

Before 2019 Microsoft's compilers could not produce UTF-8 string constants from UTF-8 source files. This is due to them converting all strings to the locale code page (which could not be UTF-8). At one time the only method to work around this was to turn ''off'' , and ''not'' mark the input file as being UTF-8 (i.e. do not use a BOM).UTF-8 Everywhere FAQ: How do I write UTF-8 string literal in my C++ code?
(note that the proposed solution does not work, string is still mangled)
This would make the compiler think both the input and outputs were in the same single-byte locale, and leave strings unmolested.


See also

*
Bush hid the facts "Bush hid the facts" is a common name for a bug present in Microsoft Windows which causes text encoded in ASCII to be interpreted as if it were UTF-16LE, resulting in garbled text. When the string "Bush hid the facts", without quotes, was put ...
, a text encoding
mojibake Mojibake (; , 'character transformation') is the garbled or gibberish text that is the result of text being decoded using an unintended character encoding. The result is a systematic replacement of symbols with completely unrelated ones, often ...


Notes


References


External links

* {{cite web , url=http://msdn.microsoft.com/en-us/library/dd374081(v=vs.85).aspx , title=Unicode , work=
MSDN Microsoft Developer Network (MSDN) was the division of Microsoft responsible for managing the firm's relationship with developers and testers, such as hardware developers interested in the operating system (OS), and software developers developing ...
, publisher=
Microsoft Microsoft Corporation is an American multinational corporation and technology company, technology conglomerate headquartered in Redmond, Washington. Founded in 1975, the company became influential in the History of personal computers#The ear ...
, access-date=November 10, 2016
Unicode Unicode or ''The Unicode Standard'' or TUS is a character encoding standard maintained by the Unicode Consortium designed to support the use of text in all of the world's writing systems that can be digitized. Version 16.0 defines 154,998 Char ...
Microsoft Windows Windows is a Product lining, product line of Proprietary software, proprietary graphical user interface, graphical operating systems developed and marketed by Microsoft. It is grouped into families and subfamilies that cater to particular sec ...