Python 3 and Unicode: Strings Are Text, Not Bytes

One of the most significant changes in Python 3 was the strict separation of text (str) and bytes (bytes). In Python 2, strings were bytes with ambiguous encoding semantics. Python 3 makes text handling explicit and Unicode-first — a design that took some adjustment but produces more correct and portable code.

str is Unicode

In Python 3, str is a sequence of Unicode code points. A string like "hello" or "こんにちは" is always Unicode text, regardless of the source file encoding. Internally, CPython uses one of three representations (Latin-1, UCS-2, or UCS-4) depending on the highest code point in the string, transparently to the programmer.

Encoding and Decoding

The str.encode() method converts text to bytes using a specified encoding: "hello".encode("utf-8") returns b'hello'. The inverse, bytes.decode(), converts bytes back to text: b'\xe2\x82\xac'.decode("utf-8") returns '€'. When opening files, always specify an encoding explicitly: open("file.txt", encoding="utf-8"). Relying on the platform default encoding leads to bugs on systems with different defaults.

Unicode Normalization in Python

Python's unicodedata module provides the normalize() function for converting between normalization forms: unicodedata.normalize("NFC", text). The module also exposes character properties like name, category, and combining class — the same properties visible in our character browser.

The Encoding Declaration

Python 3 source files default to UTF-8 encoding. If you need a different encoding (rare), add a magic comment: # -*- coding: latin-1 -*-. Python 2 required this comment to use non-ASCII characters in source files; Python 3 removed this requirement for UTF-8. Use our text encoder to verify the UTF-8 byte sequences for any Unicode string you plan to use in your source code.