| ▲ | schmorptron 4 hours ago | |
I'm not the author, but I would like! Is it feasible to run on the same hardware as the 27b model itself? | ||
| ▲ | lwarfield 4 hours ago | parent | next [-] | |
The using a J lens is super cheap compared to inference. You basically add a single matrix multiply per layer. You probably wouldn't even notice the overhead in a good implementation. You should even be able to create a j lens from scratch, but it might take a while. I was able to do it in a few hours on an H100. Creating the J lens is basically the equivalent of calculating a few thousand training steps for a model (256,000 backprops in my case). I've got more details in a blog post: https://blog.lwarfield.dev/layer-scope/ I'm currently at work and can't those matrixes up until I get home. I'll update this comment with a link later. | ||
| ▲ | jasonjmcghee 4 hours ago | parent | prev [-] | |
Fwiw you can just Google this for a model and often someone has done it | ||