The standard format for RSA private keys is ASN.1 (https://www.rfc-editor.org/rfc/rfc8017#appendix-C) with the components encoded as INTEGERs. An INTEGER is always signed in ASN.1, so you need the leading 0 byte if the MSB of your positive number is set.

OpenSSL is just dumping the raw bytes comprising the value. Tools that don't show a leading zero in this case are doing a bit more interpretation (or just treating it as an unsigned value) to show you the number you expect.

Java BigInteger is signed.

https://docs.oracle.com/javase/8/docs/api/java/math/BigInteg...

You deserialize with a leading 0x80 you don't get a working key. You can check the bit or you can just prepend a 0x00.

> Okay, so why isn't it done consistently? Some tools report the leading 00 and some don't.

This is probably a bug (where an unsigned integer with its high bit set is not printed with a leading 00) and should be reported.

Note that RSA key moduli generated by OpenSSL will always have the high bit set, and so will always have 00 prepended when you ask it to print them. The same is not necessarily true of other integers.

This is trivial to demonstrate:

    $ while true ; do openssl genrsa 1024 2>/dev/null | openssl rsa -text 2>/dev/null | grep -A1 modulus | tail -n1 | egrep -v '^\s*00:[89abcdef]' ; done

Everyone knows that an RSA modulus is a very large unsigned number yes. Not everyone knows that every number is unsigned.

> Is there some arbitrary precision library in use that forces large integers to be signed?

OpenSSL's own BN (BigNum) library, which tests if the high bit is set in the input (line 482):

https://github.com/openssl/openssl/blob/a0d1af6574ae6a0e3872...

> Even if it were signed, or had the MSB set, it wouldn't change any of the bits, so the key would still be the same. So why would we care about the sign?

Because the encoding doesn't care about the context. RFC 3279 specifies that the modulus and exponent are encoded as INTEGERs:

https://datatracker.ietf.org/doc/html/rfc3279#section-2.3.1

... and INTEGERs are signed (which means OpenSSL has to use signedness == SIGNED):

https://learn.microsoft.com/en-us/windows/win32/seccertenrol...

    Integer values are encoded into a TLV triplet that begins with a Tag
    value of 0x02.  The Value field of the TLV triplet contains the encoded
    integer if it is positive, or its two's complement if it is negative.
    If the integer is positive but the high order bit is set to 1, a leading
    0x00 is added to the content to indicate that the number is not negative.

This is exactly the same way that signed integers are represented in e.g. x86 (minus the leading tag and length fields) -- if the leading bit is set, the number is negative.

You're right that it wouldn't change any of the key's bits, but it would change the math performed on them, in a manner that would break it.