Remix.run Logo
▲ k0morebi 3 hours ago

Is my intuition correct in that the headline tok/s are for a single user?

Multiple concurrent users make that tok/s/user at best with probably additional staking penalties with each user? I see people saying this isn't necessary for sub 100 employee businesses, but how does that work out if you have 100 users?

▲osamagirl69 3 hours ago | parent | next [-]

The chart of the actual tok/s vs concurrency is at the end. For a single user you get 107tok/s, which is about 2-3x as fast as you would get on 4x DGX spark.

The 607 tok/s is total tok/s across the 16 users, so a bit over 35tok/s per user.

▲ 3 hours ago | parent | prev [-]
[deleted]