Yeah you could go with just Number though, and then keep almost everything a BigInteger/BigDecimal under the covers (or dynamically choose the correct class)

You could, but it would be significantly worse. Number and JsonNumber are nearly equivalent in this context, except that JsonNumber has the major advantage of being in the same sealed hierarchy as the other JSON types, allowing for nice pattern-matching (and BigDecimal suffers from the same downside). So you gain nothing - you have to convert to the desired type anyway - and lose quite a bit. That should be obvious.