I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:

  gzipt \
      --corpus data/tinyshakespeare.txt \  
      --prompt $'MENENIUS:\n' \
      --length 200 \
      ;
  
  MENENIUS:
  MtLUMSeptuttyyyxyxyxyxyvyyyxyxyxyxyvyyyxyxyxyxywyvzyxyxyx
  yyxyyyxyxyxyxyxPlyxyxyxyxyxyxyxyxyxtoxzfTUS.zxzzzyzzzvzzz
  vzzzxvzyvyxyxyxyvyxyxyxyvy--,Vdvyxyxyxyxyxyxyxyxyxxy!zFlx
  zzyyxyxyxyvyxyxyxyvyySPffuyuy

Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?

Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."

▲ maxidog 3 hours ago | parent | next [-]

Did you check MiMo correctly performed this unfamiliar task before posting this comment?

▲ networked 3 hours ago | parent [-]

I did. I read the code to make sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.

This was the main change for bzip2:

  @@ -33,19 +34,16 @@ def candidate_lengths(
       level: int = 9,
       pool: ThreadPoolExecutor | None = None,
   ) -> list[int]:
  -    """Compressed length of ``context + seq`` for each seq, sharing the context.
  +    """Compressed length of ``context + seq`` for each seq.
  
  -    Compresses ``context`` once into a ``compressobj``, then clones its encoder
  -    state per candidate and feeds only that candidate. Identical to
  -    ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
  -    match search over ``context`` happens a single time.
  +    Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
  +    snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
  +    whole block, so every candidate recompresses the full context. Threads
  +    still scale because ``bz2`` releases the GIL.
       """
  -    base = zlib.compressobj(level)
  -    head = len(base.compress(context))
  
       def length_for(seq: bytes) -> int:
  -        clone = base.copy()
  -        return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
  +        return len(bz2.compress(context + seq, level))
  
       if pool is not None:
           return list(pool.map(length_for, sequences))

▲ jeremyjh 2 hours ago | parent | prev [-]

So, you had an AI write code you don't understand, then posted output you don't understand in a comment on the internet for other humans to read?

	▲	an hour ago \| parent \| next [-]
		[deleted]
	▲	an hour ago \| parent \| prev [-]
		[deleted]