| ▲ | curuinor 4 hours ago | |
can't assume gaussian underlying distribution of the word-knowing, it's known zipfian. so you can't be doing anovas or anything of that nature because if you look up zipfian distribution's variance, you get Nature and Reality giving you the middle finger | ||
| ▲ | soVeryTired 4 hours ago | parent | next [-] | |
No way is vocab size zipfian. Word counts from a corpus follow zipf's law, but not vocab sizes themselves. Otherwise the most common vocab size would be equal to one. | ||
| ▲ | montag an hour ago | parent | prev [-] | |
Not to mention, N=1 | ||