Could somebody give a high level overview of this for me, as not a godev? It looks like Go JSON lib has support to encode native go structures in JSON, which is cool, but maybe it was bad, which is not as cool. Do I have that right?

▲

eknkc 3 days ago | parent | next [-]

Go already has a JSON parser and serializer. It kind of resembles the JS api where you push some objects into JSON.stringify and it serializes them. Or you push some string and get an object (or string etc) from JSON.parse.

The types themselves have a way to customize their own JSON conversion code. You could have a struct serialize itself to a string, an array, do weird gymnastics, whatever. The JSON module calls these custom implementations when available.

The current way of doing it is shit though. If you want to customize serialization, you need to return a json string basically. Then the serializer has to check if you actually managed to return something sane. You also have no idea if there were some JSON options. Maybe there is an indentation setting or whatever. No, you return a byte array.

Deserialization is also shit because a) again, no options. b) the parser has to send you a byte array to parse. Hey, I have this JSON string, parse it. If that JSON string is 100MB long, too bad, it has to be read completely and allocated again for you to work on because you can only accept a byte array to parse.

New API fixes these. They provide a Decoder or Encoder to you. These carry any options from top. And they also can stream data. So you can serialize your 10GB array value by value while the underlying writer writes it into disk for example. Instead of allocating all on memory first, as the older API forces you to.

There are other improvements too but the post mainly focuses on these so thats what I got from it (I havent tried the new api btw, this is all from the post so maybe I’m wrong on some points)

▲

tucnak 3 days ago | parent | next [-]

> If that JSON string is 100MB long, too bad, it has to be read completely and allocated again for you to work on because you can only accept a byte array to parse.

I was not sure whether this was the case, as `json.NewEncoder(io.Writer)` and `json.NewDecoder(io.Reader)` exist in v1, so I had checked, and guess what, you're right! Decode() actually reads the value to internal buffer before doing any marshalling in the first place. I had always assumed that it kept internal stack of some kind, for matching-parenthesis and type safety stuff within streaming context, but no, it doesn't do any of that stuff! Come think of it: it does make sense, as partial-unmarshal would be potentially devastating for incrementally-updated data structures as it would leave them to inconsistent state.

▲

stackedinserter 3 days ago | parent | prev [-]

gjson/sjson is probably for you if you need to work with 100MB JSONs.

	▲	trimethylpurine 3 days ago \| parent [-]
		This is cool. I wouldn't have thought to use Go for stuff that size.

▲

resonantjacket5 3 days ago | parent | prev | next [-]

The main issues are under the Behavior differences https://go.dev/blog/jsonv2-exp#behavior-differences

The largest problem were around behavior around nil in golang and what to convert into json and vice versa.

* The v2 will now throw an error for invalid characters outside of ut8 (before silently accepted it) which meant one had to preprocess or process again the json before sending it off to the server * the golang nil will be converted to json empty array or map (for each type). previously it was converted to json null. * json field names will be converted to golang names with case sensitivity. before it was case-insentitive and would be lowercased. this kinda caused lots of problems if the field collided. (say there's bankName and bankname in json) * omitempty was problematic as it was used for say golang amount: nil would mean omit the field in json as {} instead of { amount: null}. however it also meant that the golang amount: 0 would also be omitted as { amount: 0 } which surprising. the new omitempty will only do so for nil and empty arrays/hashmaps but no longer for 0 or false. there's a new omitzero tag for that.

▲

skywhopper 3 days ago | parent | prev [-]

Nah, the existing implementation is pretty decent, actually, but doesn’t address every use case and has some flaws that are hard or impossible to fix. But for lots of use cases it works great.

Now here’s a new implantation that addresses some of the architectural problems that made the old library structurally problematic for some use cases (streaming large JSON docs being the main one).

	▲	donatj 3 days ago \| parent [-]
		This. Having worked with it for over a decade the existing one is basically fine. Simple and clean interface. New one solves some niche problems I think were probably just best left to third party libraries,