| ▲ | malfist 6 hours ago |
| > nearly as accurate as a human book keeper Anything to avoid using the metric system. Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall. |
|
| ▲ | adamkurkiewicz 6 hours ago | parent | next [-] |
| Hey, author of the blog post here. I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts. |
|
| ▲ | infecto 6 hours ago | parent | prev | next [-] |
| It’s a service that provides an AI bookkeeper so it’s a pretty relevant metric. |
|
| ▲ | wat10000 6 hours ago | parent | prev [-] |
| You'd care if you had a human bookkeeper and you were considering replacing them with this company's AI bookkeeper. |
| |
| ▲ | altruios 6 hours ago | parent [-] | | Why would I want both a less accurate book keeper and to incur all of the liability of doing the books myself?! | | |
| ▲ | onraglanroad 6 hours ago | parent | next [-] | | It's not less accurate. As commented above, the control knew they were being tested against the machine, so made sure to be super careful. In everyday life the human is less careful, and the machine costs 1% of the human. | |
| ▲ | murderfs 6 hours ago | parent | prev | next [-] | | Presumably your current book keeper is not your slave, and you have to pay them... | | |
| ▲ | altruios 6 hours ago | parent [-] | | ...again: liability is the key issue. Cost savings a not exactly an isolated issue here. | | |
| ▲ | murderfs 6 hours ago | parent | next [-] | | Then why only have one human bookkeeper? Surely two would be better, since you can compare their results. But then, perhaps you should hire three, so you can figure out which one is right. | | |
| ▲ | altruios 5 hours ago | parent | next [-] | | Regardless of how correct they are, they assume liability: which is a metric that you do not improve with more bookkeepers. | |
| ▲ | deno 6 hours ago | parent | prev [-] | | But what if the third bookkeeper is malicious? You need at least four bookkeepers to achieve byzantine fault tolerance with f=1. | | |
| |
| ▲ | CamperBob2 5 hours ago | parent | prev [-] | | Got some bad news for you about that "liability" thing: it's always been on you. Read the fine print on your tax forms sometime. |
|
| |
| ▲ | wat10000 6 hours ago | parent | prev [-] | | Because it's way cheaper. Not saying it's a good tradeoff, but that appears to be their pitch. |
|
|
