My gripe with new languages like this is that strings are always UTF-8 and that makes it needlessly difficult to parse HTTP headers correctly. This has led to vulnerabilities in the past.

▲

josephg 3 days ago | parent | next [-]

I disagree. UTF-8 is the right default. Pure ASCII strings are rarely needed in modern software. Unless you know that you need ASCII, your strings should support unicode.

▲

tosti 2 days ago | parent [-]

Internet Messages have been ASCII since Internet Messages.

▲

jcranmer 2 days ago | parent | next [-]

Most internet protocols these days have shifted to UTF-8.

	▲	tosti 2 days ago \| parent [-]
		Perhaps so, and that's a good development imho. However, the most notorious of all, HTTP, has not afaik. Different encodings have been suggested including UTF-8. I'm not entirely sure why those proposals weren't implemented, but the dot-com bubble at the time might've had something to do with it.

▲

josephg 2 days ago | parent | prev [-]

True, but humans have been speaking non English languages since humans.

▲

tosti 2 days ago | parent [-]

The first networked computers predate unicode by several decades. Back then, the word length could be 8 but not neccesarily. All sorts of different encodings existed and a given common standard wasn't yet agreed upon

It's not like internet standards don't know there are other languages, it's just that they documented how things were done at the time. Some legacy has remained ever since.

▲

josephg 2 days ago | parent [-]

Sure, but most code doesn't interact with raw packets. Legacy, ascii-only internet standards probably make up much less than 1% of new lines of code written today. Programming languages should support this use case. But not at the expense of the other 99% of software.

Unicode-aware strings are the right choice for 99% of code. The last 1% should be a special case.

▲

tosti 2 days ago | parent [-]

Raw packets? Legacy? Have you've ever talked to anything over a socket?

You don't need a raw socket to get into trouble. You also don't need a "legacy" protocol.

	▲	josephg a day ago \| parent [-]
		Yes, plenty of times. But I’m usually using raw byte arrays via protobuf or something similar. ASCII only text protocols are rare. Most devs don’t reimplement them from scratch. I’d wager most code is application code, where UTF8 strings are a great choice.

▲

bflesch 3 days ago | parent | prev [-]

I agree. At this point scripting languages should have native support for string subtypes with custom charsets (ascii/utf16) or just regexp-defined or even ENUM()'erated values.

IIRC historically on Windows, a string was UTF16, on unix it was ASCII; nowadays everywhere it's UTF8 without a way to specifically limit what goes into a string.

For example UTF8 opens the door to homoglyph attacks and various other things (RTL, spaces), and a program should be able to force a string to be ASCII-only so that these classes of problems are ruled out.

▲

tosti 3 days ago | parent | next [-]

The NT kernel predates UTF8 by at least 3 years. It originally used UCS2, which covers the basic multilingual plane of the first unicode standard.

▲

Georgelemental 3 days ago | parent | prev [-]

Homoglyph attacks and RTL work just as well in UTF-16. UTF-8 and UTF-16 are different character encodings for the same character set (Unicode).

That being said, Windows permits unpaired surrogates in its "UTF-16" strings, even though that is not actually valid UTF-16. Similarly, many Linux APIs accept arbitrary bytes, not just valid UTF-8.

	▲	bflesch 3 days ago \| parent [-]
		Of course. I was trying to say that it would be nice to limit certain variables to an ASCII charset, or ASCII+umlaut+accents without any of the 10k other UTF8 shenanigans.