| ▲ | post-it 2 hours ago | |
I think we're a breakthrough away. I don't think processing power is an issue. A 7B model has a lot of cruft (like knowledge of code syntax) that a character sim doesn't need. It should be possible to extract just the relevant parts of a larger and more powerful model and run it with the same resources you would run a general 7B model. We just don't have the ability to isolate and extract weights like that yet. | ||