new | show | ask | jobs Github

codr7 15 hours ago

JSON parser libraries in general is a black hole of suffering imo.

They're either written with a different use case in mind, or a complex mess of abstractions; often both.

It's not a very difficult problem to solve if you only write exactly what you need for your specific use case.

▲

mbac32768 14 hours ago | parent | next [-]

It's astonishing how involved a fucking modern JSON library becomes.

The once "very simple" C++ single-header JSON library by nlohmann is now

* 13 years old

* is still actively merging PRs (last one 5 hours ago)

* has 122 __million__ unit tests

Despite all this, it's self-admittedly still not the fastest possible way to parse JSON in C++. For that you might want to look into simdjson.

Don't start your own JSON parser library. Just don't. Yes you can whiteboard one that's 90% good enough in 45 minutes but that last 10% takes ten thousand man hours.

▲

kstenerud 3 hours ago | parent | next [-]

I did write one, but I needed to because the already-written data must be recoverable on a crash (to be able to recover partially written files) since this is in a crash reporter - and also the encoder needs to be async-safe.

https://github.com/kstenerud/KSCrash/blob/master/Sources/KSC...

And yeah, writing a JSON codec sucks.

So I'm in the process of replacing it with a BONJSON codec, which has the same capabilities, is still async-safe and crash resilient, and is 35x faster with less code.

https://github.com/kstenerud/ksbonjson/blob/main/library/src...

▲

vovavili 10 hours ago | parent | prev | next [-]

I am very surprised to hear the unit testing statistic. What kind of unholy edge cases would JSON parsing require to make it necessary to cover 122 million variations?

	▲	kstenerud 3 hours ago \| parent [-]
		The more speed optimizations you put in, the gnarlier the new edge cases that pop up.

▲

codr7 10 hours ago | parent | prev | next [-]

Yeah, but as long as I'm not releasing in public, I don't need to support 20 different ways of parsing.

That's the thing with reinventing wheels, a wheel that fits every possible vehicle and runs well in any possible terrain is very difficult to build. But when you know exactly what you need it's a different story.

▲

modeless 9 hours ago | parent | prev | next [-]

This may say more about C++ than JSON

	▲	0x000xca0xfe 8 hours ago \| parent [-]
		The best language to handle unusual JSON correctly would probably be Python. It has arbitrary size integers, mpmath for arbitrary precision floats and good Unicode support.

▲

EasyMark 13 hours ago | parent | prev | next [-]

Yeah I use this and I think most of friends do too :)

▲

typpilol 13 hours ago | parent | prev | next [-]

122 million unit tests? What?

▲

fHr 12 hours ago | parent | prev [-]

holy shit

▲

forty 14 hours ago | parent | prev | next [-]

Parsing JSON is a Minefield (2016)

https://seriot.ch/projects/parsing_json.html

▲

codr7 10 hours ago | parent [-]

Not if I'm also the producer.

	▲	president_zippy 6 hours ago \| parent [-]
		Finally, I have found someone who understands the purpose of using someone else's tiny header-only C library; someone who sincerely thought about it before looking for an excuse to bitch and complain.

▲

flohofwoe 15 hours ago | parent | prev | next [-]

You can't get much more 'opinion-less' than this library though. Iterate over keys and array items, identify the value type and return string-slices.

▲

IshKebab 14 hours ago | parent [-]

It also feels like only half the job to me. Reminds me of SAX "parsers" that were barely more than lexers.

▲

flohofwoe 14 hours ago | parent [-]

I mean, what else is there to do when iterating over a JSON file? Delegating number parsing and UNICODE handling to the user can be considered a feature (since I can decide on my own how expensive/robust I want this to be).

▲

skydhash 14 hours ago | parent | next [-]

That is what I like Common Lisp libraries. They are mostly about the algorithms, leaving data structures up to the user. So you make sure you got those rights before calling the function.

▲

IshKebab 12 hours ago | parent | prev [-]

Extracting the data into objects. Libraries like Serde and Pydantic do this for you. Hell the original eval() JSON loading method did that too.

▲

meindnoch 9 hours ago | parent [-]

Then you lose the ability to do streaming.

▲

IshKebab 2 hours ago | parent [-]

True, but usually you only need that if your data is so large it can't fit in memory and in that case you shouldn't be using JSON anyway. (I was in this situation once where our JSON files grew to gigabytes and we switched to SQLite which worked extremely well.)

	▲	meindnoch 22 minutes ago \| parent [-]
		Actually, you'll hit the limits of DOM-style JSON parsers as soon as your data is larger than about half the available memory, since you'd most likely want to build your own model objects from the JSON, so at some point both of them must be present in memory (unless you're able to incrementally destroy those parts of the DOM that you're done with). Anyhow, IMO a proper JSON library should offer both, in a layered approach. That is, a lower level SAX-style parser, on top of which a DOM-style API is provided as a convenience.

▲

patrickmay 11 hours ago | parent | prev | next [-]

> JSON parser libraries in general is a black hole of suffering imo.

Sexprs sitting over here, hoping for some love.

▲

nicce 14 hours ago | parent | prev | next [-]

The project advertises that it has zero-allocations with minimal state. I don’t think it is fair or our problems are very different. Single string, (the most used type), and you need an allocation.

▲

TheRealPomax 12 hours ago | parent | prev [-]

Anyone who claims "it's not a very difficult problem" hasn't actually had to solve that problem.

	▲	codr7 10 hours ago \| parent [-]
		Except I have, several times, with gopd results. So in this case you're wrong. General purpose is a different can of worms compared to solving a specific case.