As an aside on the GNU *sum tools, I found they're quite slow. A few months ago I wrote a simple replacement in Go for UX reasons and somewhat to my surprise, the Go implementation of most hash algorithms seems about 2 to 4 times as fast when using a simple naïve "obvious" single-threaded implementation. It can be sped up even more by using more than one core. Go has assembly implementations for most hash functions. I didn't really look at the coreutils implementation but I'm guessing it's "just" in C.

At any rate, small teething issues aside, long-term things should be better and faster.

▲ collinfunk 12 hours ago | parent [-]

What distribution do you use?

GNU Coreutils uses the OpenSSL implementation of hashes by default, but some distributions have disabled it using './configure --with-openssl=no'. Debian used to do this, but now links to OpenSSL.

▲ arp242 11 hours ago | parent [-]

This is on Void. It doesn't have --with-openssl configure args in the package, although the binary also doesn't link to lib{ssl,crypto}. It probably gets auto-detected to "no"(?)

▲ collinfunk 11 hours ago | parent [-]

Yep, from a Void Linux container it appears that they do not link to libcrypto:

  $ ldd /usr/bin/cksum
   linux-vdso.so.1 (0x00007fb354763000)
   libc.so.6 => /usr/lib64/libc.so.6 (0x00007fb354549000)
   /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb354765000)

For context, I am a committer to GNU Coreutils. We have used OpenSSL by default for a few years now [1]. Previously it was disabled by default because the OpenSSL license is not compatible with the GPL license [2]. This was resolved when they switched to the Apache License, Version 2.0 in OpenSSL 3.0.0.

If the Void people wanted to enable OpenSSL, they would probably just need to add openssl (or whatever they package it as) to the package dependencies.

[1] https://github.com/coreutils/coreutils/commit/0d77e1b7ea2840... [2] https://www.gnu.org/licenses/license-list.html#OpenSSL

▲ arp242 7 hours ago | parent [-]

Cheers; I guess I should have checked the coreutils implementation; I kind of just assumed it has one implementation instead of being a compile option :embarrassed-emoji:

I also have an Arch machine where it does link to libcrypto, and it seems roughly identical (or close enough that I don't care, this is a live server doing tons of $stuff so has big error bars):

  md5sum              1.58s user 0.31s system 98% cpu 1.908 total
  ~/verify -a md5     1.59s user 0.13s system 99% cpu 1.719 total
  
  sha256sum           0.71s user 0.12s system 99% cpu 0.840 total
  ~/verify -a sha256  0.74s user 0.12s system 99% cpu 0.862 total

Still wish it could do multi-core though; one reason I looked in to this is because I wanted to check 400G of files and had 15 cores doing nothing (I know GNU parallel exists, but I find it hard to use and am never quite sure I'm using it correctly, so it's faster to write my own little Go program – especially for verifying files).

▲ collinfunk 6 hours ago | parent [-]

Interesting, there must be something wrong here. Here is a benchmark using the same commit and default options other than adjusting '--with-openssl=[yes|no]':

  $ dd if=/dev/random of=input bs=1000 count=$(($(echo 10G | numfmt --from=iec) / 1000))
  10737418+0 records in
  10737418+0 records out
  10737418000 bytes (11 GB, 10 GiB) copied, 86.3693 s, 124 MB/s
  $ time ./src/sha256sum-libcrypto input 
  b3e702bb55a109bc73d7ce03c6b4d260c8f2b7f404c8979480c68bc704b64255  input

  real 0m16.022s
  $ time ./src/sha256sum-nolibcrypto input 
  b3e702bb55a109bc73d7ce03c6b4d260c8f2b7f404c8979480c68bc704b64255  input

  real 0m39.339s

Perhaps there is something wrong with the detection on your system? As in, you do not have this at the end of './configure':

  $ grep -F 'HAVE_OPENSSL_' lib/config.h
  #define HAVE_OPENSSL_MD5 1
  #define HAVE_OPENSSL_MD5_H 1
  #define HAVE_OPENSSL_SHA1 1
  #define HAVE_OPENSSL_SHA256 1
  #define HAVE_OPENSSL_SHA3 1
  #define HAVE_OPENSSL_SHA512 1
  #define HAVE_OPENSSL_SHA_H 1

	▲	arp242 4 hours ago \| parent [-]
		Sorry, I meant "roughly identical [to my Go program]", not "roughly identical [to the version without OpenSSL]". The ~/verify binary is my little Go program that is ~4 times faster on my Void system, but is of roughly equal performance on the Arch system, to check that coreutils is not slower than Go (when using OpenSSL). Sorry, I probably didn't make that too clear.