Unicode 物件與編解碼器¶

Unicode 物件¶

Since the implementation of PEP 393 in Python 3.3, Unicode objects internally use a variety of representations, in order to allow handling the complete range of Unicode characters while staying memory efficient. There are special cases for strings where all code points are below 128, 256, or 65536; otherwise, code points must be below 1114112 (which is the full Unicode range).

UTF-8 表示法會在需要時建立並快取在 Unicode 物件中。

備註

自 Python 3.12 起，已移除 Py_UNICODE 表示法，並標示為已棄用的 API。更多資訊請參閱 PEP 623。

Unicode 型別¶

這些是 Python 中用於 Unicode 實作的基本 Unicode 物件型別：

PyTypeObject PyUnicode_Type¶: 為穩定 ABI 的一部分.
This instance of PyTypeObject represents the Python Unicode type. It is exposed to Python code as str.

PyTypeObject PyUnicodeIter_Type¶: 為穩定 ABI 的一部分.
This instance of PyTypeObject represents the Python Unicode iterator type. It is used to iterate over Unicode string objects.

type Py_UCS4¶
type Py_UCS2¶
type Py_UCS1¶: 為穩定 ABI 的一部分.
These types are typedefs for unsigned integer types wide enough to contain characters of 32 bits, 16 bits and 8 bits, respectively. When dealing with single Unicode characters, use Py_UCS4.

在 3.3 版被加入.

type PyASCIIObject¶

type PyCompactUnicodeObject¶

type PyUnicodeObject¶

These subtypes of PyObject represent a Python Unicode object. In almost all cases, they shouldn't be used directly, since all API functions that deal with Unicode objects take and return PyObject pointers.

在 3.3 版被加入.

The structure of a particular object can be determined using the following macros. The macros cannot fail; their behavior is undefined if their argument is not a Python Unicode object.

PyUnicode_IS_COMPACT(o)¶: True if o uses the PyCompactUnicodeObject structure.

在 3.3 版被加入.

PyUnicode_IS_COMPACT_ASCII(o)¶: True if o uses the PyASCIIObject structure.

在 3.3 版被加入.

The following APIs are C macros and static inlined functions for fast checks and access to internal read-only data of Unicode objects:

int PyUnicode_Check(PyObject *obj)¶: Return true if the object obj is a Unicode object or an instance of a Unicode subtype. This function always succeeds.

int PyUnicode_CheckExact(PyObject *obj)¶: Return true if the object obj is a Unicode object, but not an instance of a subtype. This function always succeeds.

Py_ssize_t PyUnicode_GET_LENGTH(PyObject *unicode)¶: Return the length of the Unicode string, in code points. unicode has to be a Unicode object in the "canonical" representation (not checked).

在 3.3 版被加入.

Py_UCS1 *PyUnicode_1BYTE_DATA(PyObject *unicode)¶
Py_UCS2 *PyUnicode_2BYTE_DATA(PyObject *unicode)¶
Py_UCS4 *PyUnicode_4BYTE_DATA(PyObject *unicode)¶: Return a pointer to the canonical representation cast to UCS1, UCS2 or UCS4 integer types for direct character access. No checks are performed if the canonical representation has the correct character size; use PyUnicode_KIND() to select the right function.

在 3.3 版被加入.

PyUnicode_1BYTE_KIND¶
PyUnicode_2BYTE_KIND¶
PyUnicode_4BYTE_KIND¶: Return values of the PyUnicode_KIND() macro.

在 3.3 版被加入.

在 3.12 版的變更: PyUnicode_WCHAR_KIND 已被移除。

int PyUnicode_KIND(PyObject *unicode)¶: Return one of the PyUnicode kind constants (see above) that indicate how many bytes per character this Unicode object uses to store its data. unicode has to be a Unicode object in the "canonical" representation (not checked).

在 3.3 版被加入.

void *PyUnicode_DATA(PyObject *unicode)¶: Return a void pointer to the raw Unicode buffer. unicode has to be a Unicode object in the "canonical" representation (not checked).

在 3.3 版被加入.

void PyUnicode_WRITE(int kind, void *data, Py_ssize_t index, Py_UCS4 value)¶

Write the code point value to the given zero-based index in a string.

The kind value and data pointer must have been obtained from a string using PyUnicode_KIND() and PyUnicode_DATA() respectively. You must hold a reference to that string while calling PyUnicode_WRITE(). All requirements of PyUnicode_WriteChar() also apply.

The function performs no checks for any of its requirements, and is intended for usage in loops.

在 3.3 版被加入.

Py_UCS4 PyUnicode_READ(int kind, void *data, Py_ssize_t index)¶: Read a code point from a canonical representation data (as obtained with PyUnicode_DATA()). No checks or ready calls are performed.

在 3.3 版被加入.

Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index)¶: Read a character from a Unicode object unicode, which must be in the "canonical" representation. This is less efficient than PyUnicode_READ() if you do multiple consecutive reads.

在 3.3 版被加入.

Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *unicode)¶: Return the maximum code point that is suitable for creating another string based on unicode, which must be in the "canonical" representation. This is always an approximation but more efficient than iterating over the string.

在 3.3 版被加入.

int PyUnicode_IsIdentifier(PyObject *unicode)¶: 為穩定 ABI 的一部分.
Return 1 if the string is a valid identifier according to the language definition, section Names (identifiers and keywords). Return 0 otherwise.

在 3.9 版的變更: The function does not call Py_FatalError() anymore if the string is not ready.

unsigned int PyUnicode_IS_ASCII(PyObject *unicode)¶: Return true if the string only contains ASCII characters. Equivalent to str.isascii().

在 3.2 版被加入.

Unicode Character Properties¶

Unicode provides many different character properties. The most often needed ones are available through these macros which are mapped to C functions depending on the Python configuration.

int Py_UNICODE_ISSPACE(Py_UCS4 ch)¶: 根據 ch 是否為空白字元來回傳 1 或 0。

int Py_UNICODE_ISLOWER(Py_UCS4 ch)¶: 根據 ch 是否為小寫字元來回傳 1 或 0。

int Py_UNICODE_ISUPPER(Py_UCS4 ch)¶: 根據 ch 是否為大寫字元來回傳 1 或 0。

int Py_UNICODE_ISTITLE(Py_UCS4 ch)¶: 根據 ch 是否為首字大寫字元來回傳 1 或 0。

int Py_UNICODE_ISLINEBREAK(Py_UCS4 ch)¶: 根據 ch 是否為換行字元來回傳 1 或 0。

int Py_UNICODE_ISDECIMAL(Py_UCS4 ch)¶: 根據 ch 是否為十進位字元來回傳 1 或 0。

int Py_UNICODE_ISDIGIT(Py_UCS4 ch)¶: 根據 ch 是否為數字 (digit) 字元來回傳 1 或 0。

int Py_UNICODE_ISNUMERIC(Py_UCS4 ch)¶: 根據 ch 是否為數值 (numeric) 字元來回傳 1 或 0。

int Py_UNICODE_ISALPHA(Py_UCS4 ch)¶: 根據 ch 是否為字母字元來回傳 1 或 0。

int Py_UNICODE_ISALNUM(Py_UCS4 ch)¶: 根據 ch 是否為字母數字 (alphanumeric) 字元來回傳 1 或 0。

int Py_UNICODE_ISPRINTABLE(Py_UCS4 ch)¶: 根據 ch 是否為可列印字元（如 str.isprintable() 所定義）來回傳 1 或 0。

這些 API 可用於快速直接字元轉換：

Py_UCS4 Py_UNICODE_TOLOWER(Py_UCS4 ch)¶: 回傳轉換為小寫的 ch 字元。

Py_UCS4 Py_UNICODE_TOUPPER(Py_UCS4 ch)¶: 回傳轉換為大寫的 ch 字元。

Py_UCS4 Py_UNICODE_TOTITLE(Py_UCS4 ch)¶: 回傳轉換為首字大寫的 ch 字元。

int Py_UNICODE_TODECIMAL(Py_UCS4 ch)¶: 回傳轉換為十進位正整數的 ch 字元，若無法轉換則回傳 -1。此函式不會引發例外。

int Py_UNICODE_TODIGIT(Py_UCS4 ch)¶: 回傳轉換為單一數字整數的 ch 字元，若無法轉換則回傳 -1。此函式不會引發例外。

double Py_UNICODE_TONUMERIC(Py_UCS4 ch)¶: 回傳轉換為雙精度浮點數 (double) 的 ch 字元，若無法轉換則回傳 -1.0。此函式不會引發例外。

這些 API 可用於處理代理字元：

int Py_UNICODE_IS_SURROGATE(Py_UCS4 ch)¶: 檢查 ch 是否為代理字元 (surrogate, 0xD800 <= ch <= 0xDFFF)。

int Py_UNICODE_IS_HIGH_SURROGATE(Py_UCS4 ch)¶: 檢查 ch 是否為高代理字元 (high surrogate, 0xD800 <= ch <= 0xDBFF)。

int Py_UNICODE_IS_LOW_SURROGATE(Py_UCS4 ch)¶: 檢查 ch 是否為低代理字元 (low surrogate, 0xDC00 <= ch <= 0xDFFF)。

Py_UCS4 Py_UNICODE_HIGH_SURROGATE(Py_UCS4 ch)¶: Return the high UTF-16 surrogate (0xD800 to 0xDBFF) for a Unicode code point in the range [0x10000; 0x10FFFF].

Py_UCS4 Py_UNICODE_LOW_SURROGATE(Py_UCS4 ch)¶: Return the low UTF-16 surrogate (0xDC00 to 0xDFFF) for a Unicode code point in the range [0x10000; 0x10FFFF].

Py_UCS4 Py_UNICODE_JOIN_SURROGATES(Py_UCS4 high, Py_UCS4 low)¶: Join two surrogate code points and return a single Py_UCS4 value. high and low are respectively the leading and trailing surrogates in a surrogate pair. high must be in the range [0xD800; 0xDBFF] and low must be in the range [0xDC00; 0xDFFF].

Creating and accessing Unicode strings¶

To create Unicode objects and access their basic sequence properties, use these APIs:

PyObject *PyUnicode_New(Py_ssize_t size, Py_UCS4 maxchar)¶

回傳值：新的參照。

Create a new Unicode object. maxchar should be the true maximum code point to be placed in the string. As an approximation, it can be rounded up to the nearest value in the sequence 127, 255, 65535, 1114111.

On error, set an exception and return NULL.

After creation, the string can be filled by PyUnicode_WriteChar(), PyUnicode_CopyCharacters(), PyUnicode_Fill(), PyUnicode_WRITE() or similar. Since strings are supposed to be immutable, take care to not “use” the result while it is being modified. In particular, before it's filled with its final contents, a string:

must not be hashed,
must not be converted to UTF-8, or another non-"canonical" representation,
must not have its reference count changed,
must not be shared with code that might do one of the above.

This list is not exhaustive. Avoiding these uses is your responsibility; Python does not always check these requirements.

To avoid accidentally exposing a partially-written string object, prefer using the PyUnicodeWriter API, or one of the PyUnicode_From* functions below.

在 3.3 版被加入.

PyObject *PyUnicode_FromKindAndData(int kind, const void *buffer, Py_ssize_t size)¶

回傳值：新的參照。

Create a new Unicode object with the given kind (possible values are PyUnicode_1BYTE_KIND etc., as returned by PyUnicode_KIND()). The buffer must point to an array of size units of 1, 2 or 4 bytes per character, as given by the kind.

If necessary, the input buffer is copied and transformed into the canonical representation. For example, if the buffer is a UCS4 string (PyUnicode_4BYTE_KIND) and it consists only of codepoints in the UCS1 range, it will be transformed into UCS1 (PyUnicode_1BYTE_KIND).

在 3.3 版被加入.

PyObject *PyUnicode_FromStringAndSize(const char *str, Py_ssize_t size)¶

回傳值：新的參照。 為穩定 ABI 的一部分.

Create a Unicode object from the char buffer str. The bytes will be interpreted as being UTF-8 encoded. The buffer is copied into the new object. The return value might be a shared object, i.e. modification of the data is not allowed.

此函式在以下情況下會引發 SystemError：

size < 0，
str 為 NULL 且 size > 0

在 3.12 版的變更: str == NULL 且 size > 0 不再被允許。

PyObject *PyUnicode_FromString(const char *str)¶: 回傳值：新的參照。 為穩定 ABI 的一部分.
Create a Unicode object from a UTF-8 encoded null-terminated char buffer str.

PyObject *PyUnicode_FromFormat(const char *format, ...)¶

回傳值：新的參照。 為穩定 ABI 的一部分.

Take a C printf()-style format string and a variable number of arguments, calculate the size of the resulting Python Unicode string and return a string with the values formatted into it. The variable arguments must be C types and must correspond exactly to the format characters in the format ASCII-encoded string.

A conversion specifier contains two or more characters and has the following components, which must occur in this order:

The '%' character, which marks the start of the specifier.
Conversion flags (optional), which affect the result of some conversion types.
Minimum field width (optional). If specified as an '*' (asterisk), the actual width is given in the next argument, which must be of type int, and the object to convert comes after the minimum field width and optional precision.
Precision (optional), given as a '.' (dot) followed by the precision. If specified as '*' (an asterisk), the actual precision is given in the next argument, which must be of type int, and the value to convert comes after the precision.
Length modifier (optional).
Conversion type.

The conversion flag characters are:

旗標	含義
`0`	The conversion will be zero padded for numeric values.
`-`	The converted value is left adjusted (overrides the `0` flag if both are given).

The length modifiers for following integer conversions (d, i, o, u, x, or X) specify the type of the argument (int by default):

Modifier	Types
`l`	long 或 unsigned long
`ll`	long long 或 unsigned long long
`j`	`intmax_t` 或 `uintmax_t`
`z`	`size_t` 或 `ssize_t`
`t`	`ptrdiff_t`

The length modifier l for following conversions s or V specify that the type of the argument is const wchar_t*.

The conversion specifiers are:

Conversion Specifier	Type	Comment
`%`	n/a	字面 `%` 字元。
`d`, `i`	Specified by the length modifier	一個有符號 C 整數的十進位表示法。
`u`	Specified by the length modifier	一個無符號 C 整數的十進位表示法。
`o`	Specified by the length modifier	一個無符號 C 整數的八進位表示法。
`x`	Specified by the length modifier	一個無符號 C 整數的十六進位表示法（小寫）。
`X`	Specified by the length modifier	一個無符號 C 整數的十六進位表示法（大寫）。
`c`	int	一個單一字元。
`s`	const char* 或 const wchar_t*	一個以 null 結尾的 C 字元陣列。
`p`	const void*	The hex representation of a C pointer. Mostly equivalent to `printf("%p")` except that it is guaranteed to start with the literal `0x` regardless of what the platform's `printf` yields.
`A`	PyObject*	呼叫 `ascii()` 的結果。
`U`	PyObject*	一個 Unicode 物件。
`V`	PyObject、const char 或 const wchar_t*	A Unicode object (which may be `NULL`) and a null-terminated C character array as a second parameter (which will be used, if the first parameter is `NULL`).
`S`	PyObject*	呼叫 `PyObject_Str()` 的結果。
`R`	PyObject*	呼叫 `PyObject_Repr()` 的結果。
`T`	PyObject*	Get the fully qualified name of an object type; call `PyType_GetFullyQualifiedName()`.
`#T`	PyObject*	Similar to `T` format, but use a colon (`:`) as separator between the module name and the qualified name.
`N`	PyTypeObject*	Get the fully qualified name of a type; call `PyType_GetFullyQualifiedName()`.
`#N`	PyTypeObject*	Similar to `N` format, but use a colon (`:`) as separator between the module name and the qualified name.

備註

The width formatter unit is number of characters rather than bytes. The precision formatter unit is number of bytes or wchar_t items (if the length modifier l is used) for "%s" and "%V" (if the PyObject* argument is NULL), and a number of characters for "%A", "%U", "%S", "%R" and "%V" (if the PyObject* argument is not NULL).

備註

Unlike to C printf() the 0 flag has effect even when a precision is given for integer conversions (d, i, u, o, x, or X).

在 3.2 版的變更: 新增對 "%lld" 和 "%llu" 的支援。

在 3.3 版的變更: 新增對 "%li"、"%lli" 和 "%zi" 的支援。

在 3.4 版的變更: 新增對 "%s"、"%A"、"%U"、"%V"、"%S"、"%R" 的寬度和精確度格式化支援。

在 3.12 版的變更: Support for conversion specifiers o and X. Support for length modifiers j and t. Length modifiers are now applied to all integer conversions. Length modifier l is now applied to conversion specifiers s and V. Support for variable width and precision *. Support for flag -.

An unrecognized format character now sets a SystemError. In previous versions it caused all the rest of the format string to be copied as-is to the result string, and any extra arguments discarded.

在 3.13 版的變更: 新增對 %T、%#T、%N 和 %#N 格式的支援。

PyObject *PyUnicode_FromFormatV(const char *format, va_list vargs)¶: 回傳值：新的參照。 為穩定 ABI 的一部分.
Identical to PyUnicode_FromFormat() except that it takes exactly two arguments.

PyObject *PyUnicode_FromObject(PyObject *obj)¶

回傳值：新的參照。 為穩定 ABI 的一部分.

Copy an instance of a Unicode subtype to a new true Unicode object if necessary. If obj is already a true Unicode object (not a subtype), return a new strong reference to the object.

Objects other than Unicode or its subtypes will cause a TypeError.

PyObject *PyUnicode_FromOrdinal(int ordinal)¶

回傳值：新的參照。 為穩定 ABI 的一部分.

Create a Unicode Object from the given Unicode code point ordinal.

The ordinal must be in range(0x110000). A ValueError is raised in the case it is not.

PyObject *PyUnicode_FromEncodedObject(PyObject *obj, const char *encoding, const char *errors)¶

回傳值：新的參照。 為穩定 ABI 的一部分.

Decode an encoded object obj to a Unicode object.

bytes, bytearray and other bytes-like objects are decoded according to the given encoding and using the error handling defined by errors. Both can be NULL to have the interface use the default values (see 內建編解碼器 for details).

All other objects, including Unicode objects, cause a TypeError to be set.

The API returns NULL if there was an error. The caller is responsible for decref'ing the returned objects.

void PyUnicode_Append(PyObject **p_left, PyObject *right)¶

為穩定 ABI 的一部分.

Append the string right to the end of p_left. p_left must point to a strong reference to a Unicode object; PyUnicode_Append() releases ("steals") this reference.

於錯誤發生時，將 *p_left 設為 NULL 並設定例外。

On success, set *p_left to a new strong reference to the result.

void PyUnicode_AppendAndDel(PyObject **p_left, PyObject *right)¶: 為穩定 ABI 的一部分.
The function is similar to PyUnicode_Append(), with the only difference being that it decrements the reference count of right by one.

PyObject *PyUnicode_BuildEncodingMap(PyObject *string)¶: 回傳值：新的參照。 為穩定 ABI 的一部分.
Return a mapping suitable for decoding a custom single-byte encoding. Given a Unicode string string of up to 256 characters representing an encoding table, returns either a compact internal mapping object or a dictionary mapping character ordinals to byte values. Raises a TypeError and return NULL on invalid input.

在 3.2 版被加入.

const char *PyUnicode_GetDefaultEncoding(void)¶

為穩定 ABI 的一部分.

Return the name of the default string encoding, "utf-8". See sys.getdefaultencoding().

The returned string does not need to be freed, and is valid until interpreter shutdown.

Py_ssize_t PyUnicode_GetLength(PyObject *unicode)¶

為穩定 ABI 的一部分自 3.7 版本開始.

Return the length of the Unicode object, in code points.

發生錯誤時，設定例外並回傳 -1。

在 3.3 版被加入.

Py_ssize_t PyUnicode_CopyCharacters(PyObject *to, Py_ssize_t to_start, PyObject *from, Py_ssize_t from_start, Py_ssize_t how_many)¶

Copy characters from one Unicode object into another. This function performs character conversion when necessary and falls back to memcpy() if possible. Returns -1 and sets an exception on error, otherwise returns the number of copied characters.

The string must not have been “used” yet. See PyUnicode_New() for details.

在 3.3 版被加入.

int PyUnicode_Resize(PyObject **unicode, Py_ssize_t length);¶

為穩定 ABI 的一部分.

Resize a Unicode object *unicode to the new length in code points.

Try to resize the string in place (which is usually faster than allocating a new string and copying characters), or create a new string.

*unicode is modified to point to the new (resized) object and 0 is returned on success. Otherwise, -1 is returned and an exception is set, and *unicode is left untouched.

The function doesn't check string content, the result may not be a string in canonical representation.

Py_ssize_t PyUnicode_Fill(PyObject *unicode, Py_ssize_t start, Py_ssize_t length, Py_UCS4 fill_char)¶

Fill a string with a character: write fill_char into unicode[start:start+length].

Fail if fill_char is bigger than the string maximum character, or if the string has more than 1 reference.

The string must not have been “used” yet. See PyUnicode_New() for details.

Return the number of written character, or return -1 and raise an exception on error.

在 3.3 版被加入.

int PyUnicode_WriteChar(PyObject *unicode, Py_ssize_t index, Py_UCS4 character)¶

為穩定 ABI 的一部分自 3.7 版本開始.

Write a character to the string unicode at the zero-based index. Return 0 on success, -1 on error with an exception set.

This function checks that unicode is a Unicode object, that the index is not out of bounds, and that the object's reference count is one. See PyUnicode_WRITE() for a version that skips these checks, making them your responsibility.

The string must not have been “used” yet. See PyUnicode_New() for details.

在 3.3 版被加入.

Py_UCS4 PyUnicode_ReadChar(PyObject *unicode, Py_ssize_t index)¶

為穩定 ABI 的一部分自 3.7 版本開始.

Read a character from a string. This function checks that unicode is a Unicode object and the index is not out of bounds, in contrast to PyUnicode_READ_CHAR(), which performs no error checking.

成功時回傳字元，發生錯誤時設定例外並回傳 -1。

在 3.3 版被加入.

PyObject *PyUnicode_Substring(PyObject *unicode, Py_ssize_t start, Py_ssize_t end)¶: 回傳值：新的參照。 為穩定 ABI 的一部分自 3.7 版本開始.
Return a substring of unicode, from character index start (included) to character index end (excluded). Negative indices are not supported. On error, set an exception and return NULL.

在 3.3 版被加入.

Py_UCS4 *PyUnicode_AsUCS4(PyObject *unicode, Py_UCS4 *buffer, Py_ssize_t buflen, int copy_null)¶: 為穩定 ABI 的一部分自 3.7 版本開始.
Copy the string unicode into a UCS4 buffer, including a null character, if copy_null is set. Returns NULL and sets an exception on error (in particular, a SystemError if buflen is smaller than the length of unicode). buffer is returned on success.

在 3.3 版被加入.

Py_UCS4 *PyUnicode_AsUCS4Copy(PyObject *unicode)¶: 為穩定 ABI 的一部分自 3.7 版本開始.
Copy the string unicode into a new UCS4 buffer that is allocated using PyMem_Malloc(). If this fails, NULL is returned with a MemoryError set. The returned buffer always has an extra null code point appended.

在 3.3 版被加入.

Locale Encoding¶

The current locale encoding can be used to decode text from the operating system.

PyObject *PyUnicode_DecodeLocaleAndSize(const char *str, Py_ssize_t length, const char *errors)¶

回傳值：新的參照。 為穩定 ABI 的一部分自 3.7 版本開始.

Decode a string from UTF-8 on Android and VxWorks, or from the current locale encoding on other platforms. The supported error handlers are "strict" and "surrogateescape" (PEP 383). The decoder uses "strict" error handler if errors is NULL. str must end with a null character but cannot contain embedded null characters.

Use