There are plenty of good enough ways:

* For lossless compression of generic data, gzip or zstd.

* For text, documentation, and information without fancy formatting, markdown, which is effectively a plain-text superset.

* For small datasets, blobs, objects, and what not, JSON.

* For larger datasets and durable storage, SQLite3.

Whenever there's text involved, use UTF-8. Whenever there's dates, use ISO8601 format (UTC timezone) or Unix timestamps.

Following these rules will keep you happy 80% of the time.

▲

nicbou 4 hours ago | parent [-]

One format I'm missing: storage for conversations and social media posts. Both are complex media (text + images/videos + metadata), and one is actually a collection of such posts.

How would you go about storing those in a somewhat human-readable format? My goal is to archive my chats and social media activity.

▲

ronsor 3 hours ago | parent | next [-]

Use a SQLite3 database. Have a table for the posts (or any other appropriate schema, depending on what metadata you have). Using SQLite3 has the advantage of future flexibility (new/different tables and schema as needed, full-text search, etc.).

You can have another table for attachments (images, videos, etc.). If they're small, store them directly in a BLOB. If they're not, store them alongside the database, and only store the relative path in the attachments table.

You may opt to convert images and videos to a single format (e.g. PNG and H.264 MP4), but you can lose information depending on the target format. It may be preferable to leave them in the original (or highest quality) format.

▲

soulofmischief 4 hours ago | parent | prev [-]

Why not just use WARC and a program that can read them? Do archives need to be human-readable?

▲

ronsor 3 hours ago | parent [-]

The thing about archives is you either parse them now or parse them later. With how much JS and other crap is served in modern social media frontends, I'm not sure WARC is the best format for archiving from them.

	▲	ElectricalUnion 3 hours ago \| parent [-]
		But that is the point of WARC: otherwise, your archival method need some sort of general inteligence (ai or human behind the scenes) to store exacly what you need. With WARC (and good WARC tooling like Browsetrix-crawler) you store everything HTTP the site sent.