Remix.run Logo
Daneel_ 2 days ago

The spiciest file I've ever had to deal with was an 18TB text file with no carriage returns/line feeds (all on one line). It was a log generated by an older Nokia network appliance. I think I ended up 'head'ing the first 2MB into another file and opening that, then I could grok (not the AI) the format and go from there.

dylan604 2 days ago | parent [-]

Oof, that sounds nasty. Did it turn out to be a standard-ish formatting with a separator where you break the line after x number of separators? I really dislike having to parse a log like that before just being able to read the log

Daneel_ 2 days ago | parent [-]

From memory there was no dedicated event separator, it just went straight from the last character of the event to the first character of the timestamp of the next event. I think there was an XML payload in the event somewhere too?

Fortunately I didn't have to edit the log in-place as we were ingesting it into Splunk, so I just wrote some parsing configuration and Splunk was able to munch on it without issue.