Unicode 物件與編解碼器

Unicode 物件

Since the implementation of PEP 393 in Python 3.3, Unicode objects internally use a variety of representations, in order to allow handling the complete range of Unicode characters while staying memory efficient. There are special cases for strings where all code points are below 128, 256, or 65536; otherwise, code points must be below 1114112 (which is the full Unicode range).

UTF-8 表示法會在需要時建立並快取在 Unicode 物件中。

備註

自 Python 3.12 起，已移除 Py_UNICODE 表示法，並標示為已棄用的 API。更多資訊請參閱 PEP 623。

Unicode 型別

這些是 Python 中用於 Unicode 實作的基本 Unicode 物件型別：

PyTypeObject PyUnicode_Type

為 穩定 ABI 的一部分.

This instance of PyTypeObject represents the Python Unicode type. It is exposed to Python code as str.

PyTypeObject PyUnicodeIter_Type

為 穩定 ABI 的一部分.

This instance of PyTypeObject represents the Python Unicode iterator type. It is used to iterate over Unicode string objects.

type Py_UCS4

type Py_UCS2

type Py_UCS1

為 穩定 ABI 的一部分.

These types are typedefs for unsigned integer types wide enough to contain characters of 32 bits, 16 bits and 8 bits, respectively. When dealing with single Unicode characters, use Py_UCS4.

在 3.3 版被加入.

type PyASCIIObject

type PyCompactUnicodeObject

type PyUnicodeObject

These subtypes of PyObject represent a Python Unicode object. In almost all cases, they shouldn't be used directly, since all API functions that deal with Unicode objects take and return PyObject pointers.

在 3.3 版被加入.

The structure of a particular object can be determined using the following macros. The macros cannot fail; their behavior is undefined if their argument is not a Python Unicode object.

PyUnicode_IS_COMPACT(o)

True if o uses the PyCompactUnicodeObject structure.

在 3.3 版被加入.

PyUnicode_IS_COMPACT_ASCII(o)

True if o uses the PyASCIIObject structure.

在 3.3 版被加入.

The following APIs are C macros and static inlined functions for fast checks and access to internal read-only data of Unicode objects:

int PyUnicode_Check(PyObject *obj)

Return true if the object obj is a Unicode object or an instance of a Unicode subtype. This function always succeeds.

int PyUnicode_CheckExact(PyObject *obj)

Return true if the object obj is a Unicode object, but not an instance of a subtype. This function always succeeds.

Py_ssize_t PyUnicode_GET_LENGTH(PyObject *unicode)

Return the length of the Unicode string, in code points. unicode has to be a Unicode object in the "canonical" representation (not checked).

在 3.3 版被加入.

Py_UCS1 *PyUnicode_1BYTE_DATA(PyObject *unicode)

Py_UCS2 *PyUnicode_2BYTE_DATA(PyObject *unicode)

Py_UCS4 *PyUnicode_4BYTE_DATA(PyObject *unicode)

Return a pointer to the canonical representation cast to UCS1, UCS2 or UCS4 integer types for direct character access. No checks are performed if the canonical representation has the correct character size; use PyUnicode_KIND() to select the right function.

在 3.3 版被加入.

PyUnicode_1BYTE_KIND

PyUnicode_2BYTE_KIND

PyUnicode_4BYTE_KIND

Return values of the PyUnicode_KIND() macro.

在 3.3 版被加入.

在 3.12 版的變更: PyUnicode_WCHAR_KIND 已被移除。

int PyUnicode_KIND(PyObject *unicode)

Return one of the PyUnicode kind constants (see above) that indicate how many bytes per character this Unicode object uses to store its data. unicode has to be a Unicode object in the "canonical" representation (not checked).

在 3.3 版被加入.

void *PyUnicode_DATA(PyObject *unicode)

Return a void pointer to the raw Unicode buffer. unicode has to be a Unicode object in the "canonical" representation (not checked).

在 3.3 版被加入.

void PyUnicode_WRITE(int kind, void *data, Py_ssize_t index, Py_UCS4 value)

Write the code point value to the given zero-based index in a string.

The kind value and data pointer must have been obtained from a string using PyUnicode_KIND() and PyUnicode_DATA() respectively. You must hold a reference to that string while calling PyUnicode_WRITE(). All requirements of PyUnicode_WriteChar() also apply.

The function performs no checks for any of its requirements, and is intended for usage in loops.

在 3.3 版被加入.

Py_UCS4 PyUnicode_READ(int kind, void *data, Py_ssize_t index)

Read a code point from a canonical representation data (as obtained with PyUnicode_DATA()). No checks or ready calls are performed.

在 3.3 版被加入.

Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index)

Read a character from a Unicode object unicode, which must be in the "canonical" representation. This is less efficient than PyUnicode_READ() if you do multiple consecutive reads.

在 3.3 版被加入.

Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *unicode)

Return the maximum code point that is suitable for creating another string based on unicode, which must be in the "canonical" representation. This is always an approximation but more efficient than iterating over the string.

在 3.3 版被加入.

int PyUnicode_IsIdentifier(PyObject *unicode)

為 穩定 ABI 的一部分.

Return 1 if the string is a valid identifier according to the language definition, section Names (identifiers and keywords). Return 0 otherwise.

在 3.9 版的變更: The function does not call Py_FatalError() anymore if the string is not ready.

unsigned int PyUnicode_IS_ASCII(PyObject *unicode)

Return true if the string only contains ASCII characters. Equivalent to str.isascii().

在 3.2 版被加入.

Unicode Character Properties

Unicode provides many different character properties. The most often needed ones are available through these macros which are mapped to C functions depending on the Python configuration.

int Py_UNICODE_ISSPACE(Py_UCS4 ch)

根據 ch 是否為空白字元來回傳 1 或 0。

int Py_UNICODE_ISLOWER(Py_UCS4 ch)

根據 ch 是否為小寫字元來回傳 1 或 0。

int Py_UNICODE_ISUPPER(Py_UCS4 ch)

根據 ch 是否為大寫字元來回傳 1 或 0。

int Py_UNICODE_ISTITLE(Py_UCS4 ch)

根據 ch 是否為首字大寫字元來回傳 1 或 0。

int Py_UNICODE_ISLINEBREAK(Py_UCS4 ch)

根據 ch 是否為換行字元來回傳 1 或 0。

int Py_UNICODE_ISDECIMAL(Py_UCS4 ch)

根據 ch 是否為十進位字元來回傳 1 或 0。

int Py_UNICODE_ISDIGIT(Py_UCS4 ch)

根據 ch 是否為數字 (digit) 字元來回傳 1 或 0。

int Py_UNICODE_ISNUMERIC(Py_UCS4 ch)

根據 ch 是否為數值 (numeric) 字元來回傳 1 或 0。

int Py_UNICODE_ISALPHA(Py_UCS4 ch)

根據 ch 是否為字母字元來回傳 1 或 0。

int Py_UNICODE_ISALNUM(Py_UCS4 ch)

根據 ch 是否為字母數字 (alphanumeric) 字元來回傳 1 或 0。

int Py_UNICODE_ISPRINTABLE(Py_UCS4 ch)

根據 ch 是否為可列印字元（如 str.isprintable() 所定義）來回傳 1 或 0。

這些 API 可用於快速直接字元轉換：

Py_UCS4 Py_UNICODE_TOLOWER(Py_UCS4 ch)

回傳轉換為小寫的 ch 字元。

Py_UCS4 Py_UNICODE_TOUPPER(Py_UCS4 ch)

回傳轉換為大寫的 ch 字元。

Py_UCS4 Py_UNICODE_TOTITLE(Py_UCS4 ch)

回傳轉換為首字大寫的 ch 字元。

int Py_UNICODE_TODECIMAL(Py_UCS4 ch)

回傳轉換為十進位正整數的 ch 字元，若無法轉換則回傳 -1。此函式不會引發例外。

int Py_UNICODE_TODIGIT(Py_UCS4 ch)

回傳轉換為單一數字整數的 ch 字元，若無法轉換則回傳 -1。此函式不會引發例外。

double Py_UNICODE_TONUMERIC(Py_UCS4 ch)

回傳轉換為雙精度浮點數 (double) 的 ch 字元，若無法轉換則回傳 -1.0。此函式不會引發例外。

這些 API 可用於處理代理字元：

int Py_UNICODE_IS_SURROGATE(Py_UCS4 ch)

檢查 ch 是否為代理字元 (surrogate, 0xD800 <= ch <= 0xDFFF)。

int Py_UNICODE_IS_HIGH_SURROGATE(Py_UCS4 ch)

檢查 ch 是否為高代理字元 (high surrogate, 0xD800 <= ch <= 0xDBFF)。

int Py_UNICODE_IS_LOW_SURROGATE(Py_UCS4 ch)

檢查 ch 是否為低代理字元 (low surrogate, 0xDC00 <= ch <= 0xDFFF)。

Py_UCS4 Py_UNICODE_HIGH_SURROGATE(Py_UCS4 ch)

Return the high UTF-16 surrogate (0xD800 to 0xDBFF) for a Unicode code point in the range [0x10000; 0x10FFFF].

Py_UCS4 Py_UNICODE_LOW_SURROGATE(Py_UCS4 ch)

Return the low UTF-16 surrogate (0xDC00 to 0xDFFF) for a Unicode code point in the range [0x10000; 0x10FFFF].

Py_UCS4 Py_UNICODE_JOIN_SURROGATES(Py_UCS4 high, Py_UCS4 low)

Join two surrogate code points and return a single Py_UCS4 value. high and low are respectively the leading and trailing surrogates in a surrogate pair. high must be in the range [0xD800; 0xDBFF] and low must be in the range [0xDC00; 0xDFFF].

Creating and accessing Unicode strings

To create Unicode objects and access their basic sequence properties, use these APIs:

PyObject *PyUnicode_New(Py_ssize_t size, Py_UCS4 maxchar)

回傳值：新的參照。

Create a new Unicode object. maxchar should be the true maximum code point to be placed in the string. As an approximation, it can be rounded up to the nearest value in the sequence 127, 255, 65535, 1114111.

On error, set an exception and return NULL.

After creation, the string can be filled by PyUnicode_WriteChar(), PyUnicode_CopyCharacters(), PyUnicode_Fill(), PyUnicode_WRITE() or similar. Since strings are supposed to be immutable, take care to not “use” the result while it is being modified. In particular, before it's filled with its final contents, a string:

• must not be hashed,

• must not be converted to UTF-8, or another non-"canonical" representation,

• must not have its reference count changed,

• must not be shared with code that might do one of the above.

This list is not exhaustive. Avoiding these uses is your responsibility; Python does not always check these requirements.

To avoid accidentally exposing a partially-written string object, prefer using the PyUnicodeWriter API, or one of the PyUnicode_From* functions below.

在 3.3 版被加入.

PyObject *PyUnicode_FromKindAndData(int kind, const void *buffer, Py_ssize_t size)

回傳值：新的參照。

Create a new Unicode object with the given kind (possible values are PyUnicode_1BYTE_KIND etc., as returned by PyUnicode_KIND()). The buffer must point to an array of size units of 1, 2 or 4 bytes per character, as given by the kind.

If necessary, the input buffer is copied and transformed into the canonical representation. For example, if the buffer is a UCS4 string (PyUnicode_4BYTE_KIND) and it consists only of codepoints in the UCS1 range, it will be transformed into UCS1 (PyUnicode_1BYTE_KIND).

在 3.3 版被加入.

PyObject *PyUnicode_FromStringAndSize(const char *str, Py_ssize_t size)

回傳值：新的參照。 為 穩定 ABI 的一部分.

Create a Unicode object from the char buffer str. The bytes will be interpreted as being UTF-8 encoded. The buffer is copied into the new object. The return value might be a shared object, i.e. modification of the data is not allowed.

此函式在以下情況下會引發 SystemError：

• size < 0，

• str 為 NULL 且 size > 0

在 3.12 版的變更: str == NULL 且 size > 0 不再被允許。