| ▲ | josh-wrale an hour ago | |||||||
Sidebar: single threaded inference isn’t good enough anymore | ||||||||
| ▲ | sosodev 30 minutes ago | parent [-] | |||||||
What about do you mean by single threaded? Each token is predicted by using parallel computation on the GPU. | ||||||||
| ||||||||
| ▲ | josh-wrale an hour ago | |||||||
Sidebar: single threaded inference isn’t good enough anymore | ||||||||
| ▲ | sosodev 30 minutes ago | parent [-] | |||||||
What about do you mean by single threaded? Each token is predicted by using parallel computation on the GPU. | ||||||||
| ||||||||