Ahh, plain text, wherein "plain" does some heavy lifting. If plain text had something as simple as a 4 byte signature it could have been soooo much better.

As it is programs have to guess the following;

A) encoding. If ANSI which code page? If unicode which encoding? In the case of utf-16 big or little endian?

B) line endings? CR? LF? CRLF? I guess no-one uses LFCR... right?

C) number formats? 100.000 or 100,000? Date formats? is that mm-dd-yyyy or dd-mm-yyyy?

D) what human-language is it in?

E) CSV? Don't get me started...

Yes. Text is a lot easier to load, parse, make guesses about than say XLS. Yes all of the above things can be "guessed" to a greater or lesser extent.

Yes BOM exists to at least try solving the encoding question. Of course most files don't use them. Lots of tools don't support them. And they don't solve any of the other issues.

But sure, ASCII files in English with US date formats, and Windows line endings....no problems at all...

ASCII is just a raw binary format. It isn't just for text.

The line endings problem really isn't a problem. Pretty much every text editor out there can handle different line endings.

I don’t think number nor date formats are relevant here. For example you could have that same problem entering text into MS Word. That’s really more of an issue if you want to use text as a database rather than a document format, which isn’t really something that even plain text advocates would generally recommend.

As for human language detection, that’s a much easier problem to solve than decoding a proprietary binary blob.

CSV definitely has its warts. But it’s not like that’s the only plain text option for serialising data. (JSON, jsonlines, YAML, XML, etc). Or you could use the actual ASCII codes reserved for records, if you really wanted something that didn’t require quoting and escaping in plain text. It’s actually a pity nobody does this.

The entire point of plain text is that it doesn’t have number formats, date formats, and so on.

This is the eternal tradeoff of specificity vs. generality.

I've made a language called CSTML to do some of this: https://docs.bablr.org/guides/cstml

Basically it's a text-based language for embedding semantic metadata into some kind of underlying text stream. Is this interesting to you?

[flagged]