I process TB-size ndjson files. I want to use jq to do some simple transformations between stages of the processing pipeline (e.g. rename a field), but it so slow that I write a single-use node or rust script instead.

▲

eru 7 hours ago | parent | next [-]

This reminds me of someone who wrote a regex tool that matches by compiling regexes (at runtime of the tool) via LLVM to native code.

You could probably do something similar for a faster jq.

▲

nchmy 7 hours ago | parent | prev | next [-]

This isn't for you then

> The query language is deliberately less expressive than jq's. jsongrep is a search tool, not a transformation tool-- it finds values but doesn't compute new ones. There are no filters, no arithmetic, no string interpolation.

Mind me asking what sorts of TB json files you work with? Seems excessively immense.

▲

rennokki 5 hours ago | parent | next [-]

> Uses jq for TB json files

> Hadoop: bro

> Spark: bro

> hive: bro

> data team: bro

	▲	f311a 2 hours ago \| parent \| next [-]
		JQ is very convenient, even if your files are more than 100GB. I often need to extract one field from huge JSON line files, I just pipe jq to it to get results. It's slower, but implementing proper data processing will take more time.
	▲	anonymoushn 2 hours ago \| parent \| prev \| next [-]
		are those tools known for their fast json parsers?
	▲	3 hours ago \| parent \| prev [-]
		[deleted]

▲

szundi 6 hours ago | parent | prev [-]

[dead]

▲

messe 7 hours ago | parent | prev | next [-]

Now I'm really curious. What field are you in that ndjson files of that size are common?

I'm sure there are reasons against switching to something more efficient–we've all been there–I'm just surprised.

▲

overfeed 7 hours ago | parent [-]

> Now I'm really curious. What field are you in that ndjson files of that size are common?

I'm not OP,but structured JSON logs can easily result in humongous ndjson files, even with a modest fleet of servers over a not-very-long period of time.

▲

messe 7 hours ago | parent [-]

So what's the use case for keeping them in that format rather than something more easily indexed and queryable?

I'd probably just shove it all into Postgres, but even a multi terabyte SQLite database seems more reasonable.

▲

carlmr 6 hours ago | parent | next [-]

Replying here because the other comment is too deeply nested to reply.

Even if it's once off, some people handle a lot of once-offs, that's exactly where you need good CLI tooling to support it.

Sure jq isn't exactly super slow, but I also have avoided it in pipelines where I just need faster throughput.

rg was insanely useful in a project I once got where they had about 5GB of source files, a lot of them auto-generated. And you needed to find stuff in there. People were using Notepad++ and waiting minutes for a query to find something in the haystack. rg returned results in seconds.

	▲	messe 6 hours ago \| parent [-]
		You make some good points. I've worked in support before, so I shouldn't have discounted how frequent "once-offs" can be.

▲

paavope 7 hours ago | parent | prev [-]

The use case could be e.g. exactly processing an old trove of logs into something more easily indexed and queryable, and you might want to use jq as part of that processing pipeline

▲

messe 7 hours ago | parent [-]

Fair, but for a once-off thing performance isn't usually a major factor.

The comment I was replying to implied this was something more regular.

EDIT: why is this being downvoted? I didn't think I was rude. The person I responded to made a good point, I was just clarifying that it wasn't quite the situation I was asking about.

▲

adastra22 6 hours ago | parent | next [-]

At scale, low performance can very easily mean "longer than the lifetime of the universe to execute." The question isn't how quickly something will get done, but whether it can be done at all.

	▲	messe 5 hours ago \| parent [-]
		Good point. I said it above, but I'll repeat it here that I shouldn't have discounted how frequent once offs can be. I've worked in support before so I really should've known better

▲

bigDinosaur 6 hours ago | parent | prev [-]

Certain people/businesses deal with one-off things every day. Even for something truly one-off, if one tool is too slow it might still be the difference between being able to do it once or not at all.

▲

6 hours ago | parent | prev [-]

[deleted]