| ▲ | mitthrowaway2 5 hours ago | |
Below is just a hand-wavy argument. Neural networks are function approximators. They're great because they can be trained to approximate almost any function. What if, somewhere out there in platonic space, there's a complicated function that represents what a philosopher might call true intelligence. Perhaps it's structured totally differently than a neural network and shares no common architecture with a transformer. Transformer-based LLMs are likely completely different in their fundamental structure. But perhaps the neural networks we have today are succeeding in intellectual tasks because they are beginning to approximate it. Like a bitmap approximates a vector image, it's not the same; the neural network is merely made of a bunch of repetitive simple elements, tuned to imitate the real thing. A higher resolution bitmap image will be a better approximation but still never become a vector image no matter the pixel count. But at some point, the neural network might be good enough at intellectual tasks that when tasked with searching function space for an algorithm that does the job its neural network does but better, it finds the real thing it's been approximating all along. | ||