| ▲ | lxgr 8 hours ago |
| Word of advice to anyone considering the "minor-units precision" strategy for representing monetary amounts: Don't (or at least, don't use it as an interchange/API data format). It seems like a clever idea (fast integer math, no rounding problems for addition and subtraction), but it'll bite you incredibly hard if you ever stumble upon an edge case such as working with a partner that has a different implied number of digits for a given currency. This is especially relevant for stablecoins, which often have a different number of implied decimal digits than the "fiat" currency they represent. Also, consider representing amounts as a string type in JSON-based APIs. JSON does not specify decimal precision, so you (and all your users/vendors) will always have to make sure your parser/serializer doesn't internally lose precision by going via floating point. This can get ugly fast, and while a string seems conceptually less neat, it completely bypasses that problem. (Some will call this an anti-pattern [1], but I'd rather not fight this particular battle for ideological purity on the shoulders of my users or shareholders.) [1] https://blog.json-everything.net/posts/numbers-are-numbers-n... |
|
| ▲ | noitpmeder 6 hours ago | parent | next [-] |
| The only real correct solution here is to send mantissa and exponent as two separate integers. It's trivial to convert between exponents for whatever math you want, it can be as correct as you want, and is unambiguous. In the HFT space you save some wire space if you can commit to a consistent exponent for some {slice} up front (think instrument/tick-size/asset-class/exchange/feed/server/whatever/...) such that you only need to send the mantissa and your clients can have a hard coded exponent. However, in similar spaces it's often worth the extra uint32 to send a on-the-wire exponent such that things _can_ change and you aren't hamstrung later by earlier "we only need cents now!" design choices when, e.g., you suddenly need to support bitcoin/... prices to full precision. (your users will thank you when they don't have to coordinate a breaking change when you want to adjust your fixed exponent) |
| |
| ▲ | microgpt 6 hours ago | parent | next [-] | | If you do that though aren't you just reinventing floating-point? | | |
| ▲ | jjmarr 3 hours ago | parent [-] | | No, because you're doing decimal floating point, which eliminates the rounding errors of binary floating point. |
| |
| ▲ | lxgr 6 hours ago | parent | prev [-] | | > The only real correct solution here is to send mantissa and exponent as two separate integers. That’s essentially the same thing as a String-serialized big decimal, just less readable, no? | | |
| ▲ | gmm1990 4 hours ago | parent [-] | | That’s quite a bit slower to process. At least if you’re converting to integers to do the calculations and the calculations would be quite a bit slower if you kept the big decimal type | | |
| ▲ | lxgr 4 hours ago | parent [-] | | True, but this is usually your least concern when you're dealing with monetary amounts/math. | | |
| ▲ | mnahkies 3 hours ago | parent [-] | | They specifically mentioned HFT so I suspect they care a lot about processing speed |
|
|
|
|
|
| ▲ | antonymoose 7 hours ago | parent | prev | next [-] |
| Having done HFT / low-latency in C++ with a browser based (read: JavaScript) management front-end: Go ahead and use integer cents everyone. It’s practically an industry standard and it works just fine. Anything else is a worse compromise. |
| |
| ▲ | amluto 2 hours ago | parent | next [-] | | If someone sells you 12345.55 EUR vs USD at a rate of 1.12345, how many EUR do you think you end up with? Do you think all market participants even agree? What if the rate is 1.123456? For added fun, you can introduce division. Some systems will allow you to sell 12345.55 USD to buy EUR at a rate of 1.12345. The article’s “no lost data” tenet is not really viable when this sort of division is involved. Are you going to track your account balance is a rational number with an absolutely immense denominator forever? | |
| ▲ | notpushkin 6 hours ago | parent | prev | next [-] | | It is fine as long as you don’t cross any edge cases (crypto, or more recently stuff like AI token pricing) and don’t forget to account for third party quirks (e.g. Stripe’s zero-decimal currencies: https://docs.stripe.com/currencies#zero-decimal). | | |
| ▲ | lxgr 5 hours ago | parent [-] | | JPY not having any minor units is arguably not a “third party quirk” but just how the currency works. The same goes for various three decimal digit currencies. |
| |
| ▲ | lxgr 5 hours ago | parent | prev | next [-] | | If you’re only trading in USD and other two-decimal currencies it can work fine, yes. For anything else, it’s much worse as also detailed in TFA. | |
| ▲ | DetroitThrow 6 hours ago | parent | prev [-] | | Agree with this, working from HFT to payments to account management in the past. You can have the blockchain team be an expert in converting integer cents, or the forex team be an expert in sub-cent conversions. You don't want to require _every team_ to have expertise in float math, by default. | | |
| ▲ | lxgr 5 hours ago | parent [-] | | Big decimals are widely available and don’t require any expertise but avoid many of the footguns of implied decimal integers. |
|
|
|
| ▲ | denismenace 8 hours ago | parent | prev | next [-] |
| > but it'll bite you incredibly hard if you ever stumble upon an edge case such as working with a partner that has a different implied number of digits for a given currency Why would that be a problem? You just transform the values when interacting with their API. |
| |
| ▲ | afavour 5 hours ago | parent | next [-] | | Because a lot of the time there won’t be any error when you’re wrong, just silent data loss. | | |
| ▲ | andylynch 4 hours ago | parent [-] | | I’ve seen bugs like this in prod systems. The notional value of the error tends to make the people concerned anything but silent. |
| |
| ▲ | microgpt 6 hours ago | parent | prev | next [-] | | Customer was charged $0.995 after fees, how to represent in your data model with integer cents? | | |
| ▲ | lxgr 4 hours ago | parent | next [-] | | You'll have to decide when and how to round. Keeping individual billing items at high precision and rounding after summing them up can work; defining and documenting a rounding policy (or complying with whatever's legally required in your jurisdiction/domain) and rounding each individual billed item can as well. | |
| ▲ | denismenace an hour ago | parent | prev | next [-] | | Currency: USD
Amount: 99500
Decimals: 5 | |
| ▲ | snsnsjjsjsiisa 3 hours ago | parent | prev | next [-] | | You use 1/1000th or 1/10000th or whatever you need. You do not need “cents”. | |
| ▲ | xprnio 5 hours ago | parent | prev [-] | | Round it up | | |
| |
| ▲ | xlii 7 hours ago | parent | prev | next [-] | | Exactly, model is in integers and representation can be 1⃣3⃣ or whatever, that's why model-view separation exist. | | |
| ▲ | lxgr 7 hours ago | parent [-] | | Sure, you can do that if you can absolutely guarantee that everyone will always respect that separation and there will never be ambiguity between your internal and some partner's representation – even during incidents, even during low-level CSV-to-DB ETLs during incidents ("just one time, I promise, we don't have time to build the proper adapter, but look how similar their and our formats are"). |
| |
| ▲ | lxgr 8 hours ago | parent | prev [-] | | Sure, but are all your (and your users' and vendors') engineers and LLM agents going to remember that? When in doubt, always be explicit. | | |
| ▲ | makeitdouble 7 hours ago | parent [-] | | I'm curious how you handle that. Let's say I operate with a 4 decimal expectation and your API expects 6, is there any way to reconcile that outside of documentation and or metadata ? (which would be the same issue I guess whatever representation is used ?) | | |
| ▲ | lxgr 7 hours ago | parent [-] | | Yeah, you need to document it. Still, even if you do: Chances that your users are just going to assume you're conforming to ISO 4217, some national standard, or your competitor that they're already integrated with are pretty high, so I wouldn't take the chance. Pick something that doesn't have to be documented instead. |
|
|
|
|
| ▲ | dahart 2 hours ago | parent | prev | next [-] |
| I think I’m agreeing with you whole-heartedly if I say that article’s conclusion is at best extreme and unrealistic when it says “the parsers need to be fixed. They should support extracting any numeric type we want from JSON numbers and at any precision.” This sounds like an unreasonable position to take. “Any” is an unachievable standard that could require an unlimited engineering budget with no demonstrable value in practice. It is good to identify the lack of a standard, and to talk about what parsers do in practice, and good to discuss the gaps and unmet use-cases. It would be a good idea to suggest that there should be a more reasonable standard, perhaps. It’s just not a good idea to demand that everyone support “any” possibility when no one really needs that, no one knows what it means, and it’s not actually possible to achieve. |
|
| ▲ | gucci-on-fleek 8 hours ago | parent | prev | next [-] |
| What do you recommend instead? Standard floating-point ("float"/"double"), fixed-point arithmetic with thousandths (or smaller) of the minor unit, arbitrary-precision decimal numbers, or something else entirely? |
| |
| ▲ | lxgr 8 hours ago | parent | next [-] | | I think what matters most is your database and API representation, as well as having consistent and well-defined rounding rules. I largely agree with TFA: Round explicitly and consistently whenever you cross a boundary, i.e. database persistence and internal API calls. Use whatever works for your required business case internally (i.e. inside of procedures calculating some function of one or more input amounts). This can be regular old floats/doubles if you absolutely know what you're doing, or BigDecimal if you aren't and would rather suffer slightly slower performance than having to talk to an auditor about IEEE 754 rounding modes, or even minor-amount integers (yes, even though I just said to not use them – but you'll want to ABSOLUTELY NEVER leak them outside of your system, including your data/analytics pipeline, which might have different ideas about financial amounts than your business logic implementing a nice custom monetary type). | |
| ▲ | ivanmontillam 8 hours ago | parent | prev | next [-] | | A string type. As parent says: it completely bypasses the problem. Save the numbers between double quotes and be done with it. | | |
| ▲ | lxgr 5 hours ago | parent | next [-] | | Except that now you have a new problem: Opinionated theorists that haven’t been part of a nasty “oh no, we accidentally considered some amounts as 10x/100x/1000x larger/smaller than expected” incident in their career yet… | |
| ▲ | portly 7 hours ago | parent | prev [-] | | Storing numbers as arrays of u8? That doesn't make sense | | |
| ▲ | ivanmontillam 6 hours ago | parent | next [-] | | For JSON serialization, which doesn't support fixed-point precision it does. Floating-point precision has too many gotchas for being suitable to store Decimal types, especially for the Currency use case. | | |
| ▲ | notpushkin 6 hours ago | parent [-] | | Surely it does: {
"price": {
"amount": 1000,
"decimal_places": 2,
"currency": "USD"
}
}
| | |
| ▲ | lxgr 6 hours ago | parent [-] | | How is that better than {“amount”: “10.00”} (which also bypasses all potential floating point parsing issues that your or your counterparty’s JSON library might have)? | | |
| ▲ | jameshart 5 hours ago | parent [-] | | It is explicit about the fact that that number of decimal places is part of the data. The semantics for your string “10.00” are complex - is it considered equal to “10”? To “10.000”? To “10.001”? A user interacting with an API that uses such a string might make all sorts of assumptions about what it supports. A user interacting with an API that has an explicit decimal places concept is being told ‘decimals matter! They can vary! Here be dragons!’ | | |
| ▲ | lxgr 4 hours ago | parent [-] | | > The semantics for your string “10.00” are complex - is it considered equal to “10”? Yes, but "10 USD" would be a non-canonical representation and you probably serialized incorrectly. > To “10.000”? Yes, but same caveat as above applies. > To “10.001”? Obviously not, and any system you'd ever want to use in a financial context will tell you so. |
|
|
|
| |
| ▲ | lxgr 6 hours ago | parent | prev [-] | | It makes a lot of sense if you value correctness over performance. | | |
| ▲ | microgpt 5 hours ago | parent [-] | | Why not store them in unary then? | | |
| ▲ | lxgr 5 hours ago | parent [-] | | Unary is exactly as expressive as decimal or binary for integers, but somewhat less efficient, so why would you? | | |
| ▲ | microgpt 4 hours ago | parent [-] | | idk, why would you store integers as ASCII strings? It's somewhat less efficient. | | |
| ▲ | lxgr 3 hours ago | parent [-] | | Because it's much more explicit. Computers are fast, engineering is expensive. You usually never want to optimize prematurely when dealing with monetary amounts. |
|
|
|
|
|
| |
| ▲ | KellyCriterion 7 hours ago | parent | prev | next [-] | | Do not throw away any precision in finance/money computation, regardless what/ how you are doing it. In C# e.g., there is type decimal for those computations. | | |
| ▲ | lxgr 7 hours ago | parent [-] | | You'll definitely have to throw it away at some point. The art is in making those points well-defined and rare enough to not cause large discrepancies, but frequent enough to avoid ballooning arbitrary-precision numbers across databases and services that might not be able to handle them. | | |
| ▲ | krever 6 hours ago | parent [-] | | I really like that phrasing!
Would you mind if I steal in some form if I decide to review this part of the book? | | |
| ▲ | lxgr 6 hours ago | parent [-] | | Not at all, and thanks for writing all of this up! |
|
|
| |
| ▲ | necrotic_comp 7 hours ago | parent | prev [-] | | Floating point value stored multiplied by 10^8. That gives you a huge integer, but it's extremely accurate, especially for US denominated currencies. Easily transformed into floating point numbers for reporting/etc. | | |
|
|
| ▲ | gib444 4 hours ago | parent | prev [-] |
| What is with this Twitter esque style of discussion? Post some vague comment with no real stake in the ground, but just reply to follow ups asking for clarifications about the right way. It's exhausting. Why not put all that effort into the initial comment? Vague-posting seems to becoming more popular |
| |
| ▲ | lxgr 4 hours ago | parent [-] | | If there were a simple one-size-fits-all solution to these problems, there wouldn't be a need for a handbook, nor for a discussion, would there? I can't design everybody's systems here, but I was hoping that sharing some war stories that have cost me days or weeks of work might sensitize somebody to a few non-obvious footguns. | | |
| ▲ | gib444 4 hours ago | parent [-] | | That strawman is so large it would even scare away a human |
|
|