|< C & Win32 programming 8 | Main | ANSI, Multi-byte, Unicode and Localization 2 >|Site Index |Download |

MODULE G

CHARACTER SETS

ANSI, Multibyte, Unicode and Localization 1

What are in this Module?

1. Programming Abilities:

2. A Story of Character Sets

3. Single-byte Character Sets

4. Double-byte Character Sets

5. Unicode

6. Windows Issue

7. Surrogates

8. Windows Data Types for Strings

9. Standard C Functions

10. wsprintf() Function

11. Conventions for Function Prototypes – Win32 Programming

12. Generic Prototype

13. ANSI Prototype

14. Unicode Prototype

15. Character and Unicode strings Conversion

My Training Period: xx hours. This Module just for informational provided for whom that interested in localization.   Also, some of the information provided here is used to understand some of the convention used in Windows programming.  ANSI C, C++ have been superseded by ISO/IEC standard.  Program examples if any, compiled using Visual C++ .Net 2003.

The programming abilities for this session:

• Able to understand the variety of the character sets such as ANSI, Multibyte and Unicode.

• Able to find and collect information related to functions used for character sets.

• Able to understand and use the collected information about the functions in programs.

• Able to understand the localization (locale).

A Story of Character Sets

• A character set is a mapping of characters to their identifying code values (decimal, binary or hex). 

• Most of the character sets commonly used in computers are single-byte character sets in which each character is identified by a value one byte wide such as ASCII (American Standard Code for Information Interchange- refer to ASCII table at ASCII) code.

• For a byte (8 bits), the maximum representation is 256 (2⁸) characters, printable and non-printable.   The large number of characters in Asian languages such as Chinese, Katakana and Thailand for example led to the development of multibyte character sets, in particular the double-byte character set (DBCS).

• This is because a single byte is not enough any more to represent their scripts.  The international standard for character encoding using multibyte is Unicode standard.

• This is very useful for localization of the computer programs including the OSes.

Unicode

• Unicode is a worldwide character-encoding standard.  The development and promotion the use of the Unicode is done by Unicode Consortium (Unicode).  The ISO code is ISO/IEC 10646 and at the moment this compilation is prepared, the latest version was 4.1.0.

• Windows NT, Windows 2000, and Windows XP use it exclusively at the system level for character and string manipulation.  Unicode simplifies localization of software and improves multilingual text processing.

• By implementing it in your applications, you can enable the application with universal data exchange capabilities for global marketing, using a single binary file for every possible character code.

• Unicode defines semantics for each character, standardizes script behavior, provides a standard algorithm for bidirectional text, and defines cross-mappings to other standards.

• Among the scripts supported (but not limited) by Unicode are Latin, Greek, Han, Hiragana, and Katakana, German, French, English, Greek, Chinese, and Japanese.

• Because each Unicode code value is 16 bits wide, it is possible to have separate values for up to 65,536 (2¹⁶) characters.  Unicode-enabled functions are often referred to as "wide-character" functions.

• Note that the implementation of Unicode in 16-bit values is referred to as UTF-16.  For compatibility with 8- and 7-bit environments,UTF-8 and UTF-7 are two transformations of 16-bit Unicode values.

Windows Issue

• Windows supports applications that use either Unicode or the regular ANSI character set.  Mixed use in the same application is also possible.

• Adding Unicode support to an application is easy, and you can even maintain a single set of sources from which to compile an application that supports either Unicode or the Windows ANSI character set.

• Functions support Unicode by assigning its strings a specific data type and providing a separate set of entry points and messages to support this new data type.

• A series of macros and naming conventions make transparent migration to Unicode, or even compiling both non-Unicode and Unicode versions of an application from the same set of sources, a straightforward matter.

• Implementing Unicode as a separate data type also enables the compiler's type checking to ensure that only Unicode parameters are used with functions expecting Unicode strings.

Surrogates

• There is a need to support more characters than the 65,536 that fit in the 16-bit Unicode code space.  For example, the Chinese speaking community alone uses over 55,000 characters.

• To overcome this issue, the Unicode Standard defines surrogates.  A surrogate or surrogate pair is a pair of 16-bit Unicode code values that represent a single character.  The first (high) surrogate is a 16-bit code value in the range U+D800 to U+DBFF.

• The second (low) surrogate is a 16-bit code value in the range U+DC00 to U+DFFF. Using surrogates, Unicode can support over one million characters.

• Windows 2000 introduces support for basic input, output, and simple sorting of surrogates.

• However, not all system components are surrogate compatible.  Also, surrogates are not supported in Windows 95/98/Me.

Windows Data Types for Strings

• Most string operations for Unicode can be written by using the same logic used for handling the Windows ANSI character set, except that the basic unit of operation is a 16-bit character instead of an 8-bit byte character, so we need an extra storage of memory.

• The Windows header files provide several type definitions that make it easy to create sources that can be compiled for Unicode or the ANSI character set.

• Windows API elements that use characters are generally implemented in one of the following three formats:

1. A generic version that can be compiled for either ANSI or Unicode.

2. An ANSI version.

3. A Unicode version.

• The following example shows the method used in the Windows header files to define three sets of data types: a set of generic type definitions that can compile for either ANSI or Unicode, and two sets of specific type definitions.

• The first set of specific type definitions is for use with the existing Windows (ANSI) character set, and the other is for use with Unicode (or wide) characters.

// generic types

#ifdef UNICODE

    typedef wchar_t TCHAR;

#else

    typedef unsigned char TCHAR;

#endif

typedef TCHAR * LPTSTR, *LPTCH;

 

// 8-bit character specific

typedef unsigned char CHAR;

typedef CHAR *LPSTR, *LPCH;

 

// Unicode specific (wide characters)

typedef unsigned wchar_t WCHAR;

typedef WCHAR *LPWSTR, *LPWCH;

• The letter T in a type definition designates a generic type that can be compiled for either ANSI or Unicode.

• The letter W in a type definition designates a wide-character (Unicode) type.  For the actual implementation of this method, see the winnt.h header file.

• An application using generic data types can be compiled for Unicode simply by defining UNICODE before the include statements for the header files, or during compilation.

• To compile the code for ANSI, omit the UNICODE definition but when you see the Windows data type definition section in the previous Module of this Tutorial, these data types already defined properly, so this section just a matter of informational.

• It is best to use the generic data types, but the specific types exist for applications that require mixed types.

Standard C Functions

• The standard C run-time libraries contain wide-character versions of the ANSI string functions that begin with the letters str.

• The wide-character versions of the functions start with the letters wcs (or sometimes _wcs).  The Unicode data type is compatible with the wide-character data typewchar_t in ANSI C.  This allows access to the wide-character string functions.

• Generic functions exist for all standard C string functions.  They start with the letters _tcs and are listed in the tchar.h header file.  These functions use the generic data typesTCHAR andTCHAR*

• An application must add the following lines to its program in order to use the generic functions and compile for Unicode:

#define _UNICODE

 

#include <tchar.h>

#include <wchar.h>

• Note that both the tchar.h and wchar.h are required, and that the leading underscore on the_UNICODE variable is also required.

• The printf() function defined in tchar.h supports the same format specifications aswsprintf().  Similarly, tchar.h contains a wprintf() function, in which the format string itself is a Unicode string.

wsprintf() Function

• The formatted output function wsprintf() supports Unicode by providing the following new and changed data types in its format specifications.  These format specifications affect the way thewsprintf() function interprets the corresponding passed-in parameter.

• The data type for the output text always depends on the version of the function.  Where the data type of the passed-in parameter and of the output text do not agree, wsprintf() performs a conversion from Unicode to ANSI, or vice versa, as required.

• For the Unicode version of wsprintf(), the format string is Unicode, as is the output text.

• As normal sprintf(), the wsprintf() function formats and stores a series of characters and values in a buffer.  Any arguments are converted and copied to the output buffer according to the corresponding format specification in the format string.

• The function appends a terminating null character to the characters it writes, but the return value does not include the terminating null character in its character count.  The following Table lists thewsprintf() function information.

• The second format specification causes the second argument to be converted and copied, and so on.  If there are more arguments than format specifications, the extra arguments are ignored.  If there are not enough arguments for all of the format specifications, the results are undefined.

• A format specification has the following form:

%[-][#][0][width][.precision]type

• Each field is a single character or a number signifying a particular format option.  The type characters that appear after the last optional format field determine whether the associated argument is interpreted as a character, a string, or a number.

• The simplest format specification contains only the percent sign and a type character (for example,%s).  The optional fields control other aspects of the formatting.

• The following Table lists the optional and required fields and their meanings.

• If you have noticed, the explanation in this section is quite similar to other formatted output function such asprintf() except the format specifications used and it is for Unicode.

Conventions for Function Prototypes – Win32 Programming

• Win32 also provides function prototypes in generic, ANSI, and Unicode versions.  The generic function prototypes can be compiled to produce either ANSI or Unicode prototypes.

• As an example, all three prototypes are shown in the following code sample for the SetWindowText() function.

Generic Prototype

BOOL SetWindowText(HWND hwnd, LPCTSTR lpText);

• The header file provides the generic function name implemented as a macro:

#ifdef UNICODE

#define SetWindowText SetWindowTextW

#else

#define SetWindowText SetWindowTextA

#endif // !UNICODE

• The preprocessor expands the macro into either the ANSI or Unicode function names, depending on whether UNICODE is defined.  The letter A (ANSI) or W (wide) is added at the end of the generic function name, as appropriate.  The header file then provides ANSI and Unicode function prototypes, as shown in the following examples.

ANSI Prototype

BOOL SetWindowTextA(HWND hwnd, LPCSTR lpText);

Unicode Prototype

BOOL SetWindowTextW(HWND hwnd, LPCWSTR lpText);

• Note that the generic function prototype uses the generic LPCTSTR for the text parameter, but the ANSI prototype uses LPCSTR, and the Unicode prototype usesLPCWSTR.

• You can call the generic function in your application, and then define UNICODE when you compile the code to use the Unicode function.

• To default to the ANSI function, do not define UNICODE.  You can mix function calls by using the explicit function names ending with A and W.

• This approach applies to all functions with text arguments.  Always use a generic function prototype with the generic string and character types.

• All function names that end with an uppercase W take wide-character arguments.  Some functions exist only in wide-character versions and can be used only with the appropriate data types.

• The Requirements section in the MSDN documentation for each function provides information on the function versions implemented by the system.  If there is a line that begins with Unicode, the function has separate Unicode and ANSI versions.

• Whenever a function has a length parameter for a character string, the length should be documented as a count ofTCHAR values in the string.

• This refers to bytes for ANSI versions of the function or characters for Unicode versions.  However, functions those require or return pointers to untyped memory blocks, such as the GlobalAlloc() function, are exceptions.

Character and Unicode strings Conversion

• The MultiByteToWideChar() function can be used to map a character string to a wide-character (Unicode) string.  The character string mapped by this function is not necessarily from a multibyte character set.

• The information for this function is listed in the following Table.

• Using the MultiByteToWideChar() function incorrectly can compromise the security of your application. Calling the MultiByteToWideChar() function can easily cause a buffer overrun because the size of the in buffer equals the number of bytes in the string, while the size of the Out buffer equals the number of WCHARs.

• To avoid a buffer overrun, be sure to specify a buffer size appropriate for the data type the buffer receives.

• The lpMultiByteStr and lpWideCharStr pointers must not be the same.  If they are the same, the function fails, andGetLastError() returns the value ERROR_INVALID_PARAMETER.

• The function fails if MB_ERR_INVALID_CHARS is set and encounters an invalid character in the source string.  An invalid character is either:

1. A character that is not the default character in the source string but translates to the default character whenMB_ERR_INVALID_CHARS is not set, or

2. For DBCS strings, a character which has a lead byte but no valid trailing byte.  When an invalid character is found, and MB_ERR_INVALID_CHARS is set, the function returns 0 and sets GetLastError() with the error ERROR_NO_UNICODE_TRANSLATION.

• The WideCharToMultiByte() function can be used to map a wide-character string to a new character string.  The new character string is not necessarily from a multibyte character set.

• The following Table lists the information for WideCharToMultiByte().

• Using the WideCharToMultiByte() function incorrectly can compromise the security of your application.  Calling the WideCharToMultiByte() function can easily cause a buffer overrun because the size of the in buffer equals the number of WCHARs in the string, while the size of the Out buffer equals the number of bytes.

• To avoid a buffer overrun, be sure to specify a buffer size appropriate for the data type the buffer receives.  By the way Visual C++ .Net provides buffer overrun check during the compilation.

• For strings that require validation, such as file, resource and user names, always use the WC_NO_BEST_FIT_CHARS flag with WideCharToMultiByte().  This flag prevents the function from mapping characters to characters that appear similar but have very different semantics.

• In some cases, the semantic change can be extreme e.g., symbol for ‘∞’ (infinity) maps to 8 (eight) in some code pages.

• WC_NO_BEST_FIT_CHARS is not available on Windows 95 and NT4.  If your code must run on these platforms, you can achieve the same effect by round tripping the string using MultiByteToWideChar().  Any code point that does not round trip is a best-fit character.

• The lpMultiByteStr and lpWideCharStr pointers must not be the same.  If they are the same, the function fails, andGetLastError() returnsERROR_INVALID_PARAMETER.

• If CodePage is CP_SYMBOL and cbMultiByte is less thancchWideChar, no characters are written tolpMultiByte. Otherwise, if cbMultiByte is less than cchWideChar, cbMultiByte characters are copied to the buffer pointed to bylpMultiByte.

• An application can use the lpDefaultChar parameter to change the default character used for the conversion.

• As noted earlier, the WideCharToMultiByte() function operates most efficiently when bothlpDefaultChar and lpUsedDefaultChar are NULL.  The following table shows the behavior ofWideCharToMultiByte() for the four combinations oflpDefaultChar and lpUsedDefaultChar.

Further reading and digging:

1. Check the best selling C, C++ and Windows books at Amazon.com.

2. Microsoft C references, online MSDN.

3. Microsoft Visual C++, online MSDN.

4. ReactOS - Windows binary compatible OS - C/C++ source code repository, Doxygen.

5. Structure, enum, union and typedef story can be found at C/C++ struct, enum, union & typedef.

6. Linux Access Control Lists (ACL) info can be found atAccess Control Lists.

7. Structure, enum, union and typedef story can be foundC/C++ struct, enum, union & typedef.

8. For Unicode and character set reference that contains functions, structures, macros and constants:Unicode and character set reference (MSDN).

9. Notation used in MSDN is Hungarian Notation instead of CamelCase and is discussedWindows programming notations.

10. Windows data type information is inWindows data types used in Win32 programming.

|< C & Win32 programming 8 | Main | ANSI, Multi-byte, Unicode and Localization 2 >|Site Index |Download |