| ▲ | k0morebi 3 hours ago | |
Is my intuition correct in that the headline tok/s are for a single user? Multiple concurrent users make that tok/s/user at best with probably additional staking penalties with each user? I see people saying this isn't necessary for sub 100 employee businesses, but how does that work out if you have 100 users? | ||
| ▲ | osamagirl69 3 hours ago | parent | next [-] | |
The chart of the actual tok/s vs concurrency is at the end. For a single user you get 107tok/s, which is about 2-3x as fast as you would get on 4x DGX spark. The 607 tok/s is total tok/s across the 16 users, so a bit over 35tok/s per user. | ||
| ▲ | 3 hours ago | parent | prev [-] | |
| [deleted] | ||