> Python essentially bet on UTF-32 (with space-saving optimisations)

How so? Python3 strings are unicode and all the encoding/decoding functions default to utf-8. In practice this means all the python I write is utf-8 compatible unicode and I don't ever have to think about it.

▲ sheept 14 hours ago | parent | next [-]

UTF-32 allows for constant time character accesses, which means that mystr[i] isn't O(n). Most other languages can only provide constant time access for code units.

	▲	msl 7 hours ago \| parent [-]
		UTF-32 allows for constant time access to code points. Neither UTF-8 nor UTF-16 can do the same (there are 2 to the power of 20 valid code points, though not all are in use). While most characters might be encodable as a single code point, Python does not normalize strings, so there is no guarantee that even relatively normal characters are actually stored as single code points. Try this in Python: `s = "a\u0308" print(s) print(s[0])` You will see: `ä a`

▲ pansa2 14 hours ago | parent | prev | next [-]

> all the encoding/decoding functions default to utf-8

Languages that use UTF-8 natively don't need those functions at all. And the ones in Python aren't trivial - see, for example, `surrogateescape`.

As the sibling comment says, the only benefit of all this encoding/decoding is that it allows strings to support constant-time indexing of code points, which isn't something that's commonly needed.

	▲	laurencerowe 13 hours ago \| parent [-]
		They absolutely do because random byte strings are not valid utf8. Safe Rust requires validating bytes when converting to strings because this.

▲ cloudbonsai 11 hours ago | parent | prev [-]

Internally Python holds a string as an array of uint32. A utf-8 representation is created on demand from it (and cached). So pansa2 is basically correct [^1].

IMO, while this may not be optimal, it's far better than the more arcane choice made by other systems. For example, due to reasons only Microsoft can understand, Windows is stuck with UTF-16.

[1] Actually it's more intelligent. For example, Python automatically uses uint8 instead of uint32 for ASCII strings.

▲ zahlman 8 hours ago | parent | next [-]

There is no caching of a "utf-8 representation". You may check for example:

  >>> x = '日本語'*100000000
  >>> import time
  >>> t = time.time(); y = x.encode(); time.time() - t # takes nontrivial time
  >>> t = time.time(); y = x.encode(); time.time() - t # not cached; not any faster

Generally, the only reason this would happen implicitly is for I/O; actual operations on the string operate directly on the internal representation.

Python uses either 8, 16 or 32 bits per character according to the maximum code point found in the string; uint8 is thus used for all strings representable in Latin-1, not just "ASCII". (It does have other optimizations for ASCII strings.)

The reason for Windows being stuck with UTF-16 is quite easy to understand: backwards compatibility. Those APIs were introduced before there supplementary Unicode planes, such that "UTF-16" could be equated with UCS-2; then the surrogate-pair logic was bolted on top of that. Basically the same thing that happened in Java.

	▲	cloudbonsai 5 hours ago \| parent [-]
		> There is no caching of a "utf-8 representation". No there certainly is. This is documented in the official API documentation: `UTF-8 representation is created on demand and cached in the Unicode object. https://docs.python.org/3/c-api/unicode.html#unicode-objects` In particular, Python's Unicode object (PyUnicodeObject) contains a field named utf8. This field is populated when PyUnicode_AsUTF8AndSize() is first called and reused thereafter. You can check the exact code I'm talking about here: https://github.com/python/cpython/blob/main/Objects/unicodeo... Is it clear enough?

▲ nslsm 10 hours ago | parent | prev | next [-]

Read first paragraph here https://devblogs.microsoft.com/oldnewthing/20190830-00/?p=10...

▲ 10 hours ago | parent | prev [-]

[deleted]