| ▲ | minimaltom 3 hours ago | ||||||||||||||||
Was really cool to see yous use Engrams to cut down compute! Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup? Also, why mHC over attention residuals? | |||||||||||||||||
| ▲ | HenryNdubuaku 3 hours ago | parent [-] | ||||||||||||||||
Yes, we ablated Engrams rigorously and found that it returned world knowledge like FFN without without compute expenditure. | |||||||||||||||||
| |||||||||||||||||