| ▲ | dytyruio 4 days ago | |
> Anthropic is purposefully degrading things to save money Regardless of whether it’s to save money, it’s purposefully inaccurate: “When Claude generates text, it calculates probabilities for each possible next word, then randomly chooses a sample from this probability distribution.” I think the reason for this is that if you were to always choose the highest probable next word, you may actually always end up with the wrong answer and/or get stuck in a loop. They could sandbag their quality or rate limit, and I know they will rate limit because I’ve seen it. But, this is a race. It’s not like Microsoft being able to take in the money for years because people will keep buying Windows. AI companies can try to offer cheap service to government and college students, but brand loyalty is less important than selecting the smarter AI to help you. | ||
| ▲ | andy99 4 days ago | parent | next [-] | |
> I think the reason for this is that if you were to always choose the highest probable next word, you may actually always end up with the wrong answer and/or get stuck in a loop. No, it's just the definition of sampling at non-zero temperature. You can set T=0 to always get the most likely token. Temperature trades of consistency for variety. You can set T to zero in the API, I assume the defaults for Claude code and their chat are nonzero. | ||
| ▲ | efskap 4 days ago | parent | prev [-] | |
>or get stuck in a loop You are absolutely right! Greedy decoding does exactly that for longer seqs: https://huggingface.co/docs/transformers/generation_strategi... Interestingly DeepSeek recommends a temperature of 0 for math/coding, effectively greedy. | ||