| ▲ | spwa4 4 hours ago |
| If those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they? |
|
| ▲ | ziofill 3 hours ago | parent | next [-] |
| This is such a beautiful question. It looks like a miracle that a brain, let alone a whole body, can grow from such little data. Michael Levin has been doing a lot of research on this, you should check him out. |
| |
| ▲ | spwa4 2 hours ago | parent [-] | | The thing is ... once those nerves are operational they bring in boatloads of data, which is useful and obviously tuned and very relevant to the specific body the neural network finds itself in. |
|
|
| ▲ | ACCount39 2 hours ago | parent | prev | next [-] |
| No room - or is there? FractalDB proves that "synthetic data useful for training AI" can be produced by a very small program. There is "no room" in DNA for a full dataset, or a full snapshot of model weights. But there's plenty of room for "this one little trick cuts the data required to learn a desirable thing X in half" reproduced thousands of times for different X. Low Kolmogorov complexity, but high performance gains. |
|
| ▲ | kikoreis 2 hours ago | parent | prev | next [-] |
| Well keep in mind we receive from our mothers an entire functional cell. DNA is a bit overrated. |
|
| ▲ | derac 3 hours ago | parent | prev | next [-] |
| The data is massively compressed and pruned. Where are the input tokens in an LLM? |
|
| ▲ | dingaling911 4 hours ago | parent | prev | next [-] |
| DNA is probably the seed for the random initialization. |
| |
| ▲ | nico 3 hours ago | parent | next [-] | | This is actually a genius idea. Imagine being able to compress a 1T parameter model to 10M params by creating a lossy model-scaffolding, like DNA Something that more or less guarantees you’ll get about the same capabilities as the original model, but with a different final expression of certain characteristics I wonder if this could be a good distribution model too, selling “dried” models (almost like raisins - or the beings from the 3 body problem), and then you can buy them to fully train (raise?) with your own data | | |
| ▲ | spwa4 2 hours ago | parent | next [-] | | Aside from violating everything we know about neural networks (technically this would mean there is a way to reduce the rank of even something like ChatGPT to 1. There isn't. No way) it would also be VERY unlike what DNA does. DNA has lots of ways to express things that aren't easily disturbed by mistakes. Those mistakes are then an actual positive, in the very long term, because they help natural selection along. A random seed that works would be the exact opposite of that. Why wouldn't DNA just encode "good feelings" when IT gets what it wants? Right temperature -> ahhh. Food -> ahhhhhh. Sex -> aaaaaahhhhhhhhhh. And then just let reinforcement learning take care of all the details in between. No need for storage space for fire making or for "Hey girl, how are you doing?" ... and therefore cheaper. | |
| ▲ | dingaling911 an hour ago | parent | prev [-] | | I'm somewhat of a genius (self-described.) I like your idea of "base" models for sale. |
| |
| ▲ | wernam 3 hours ago | parent | prev [-] | | It's nice to throw around claims like that, or like "microtubules connect the brain the quantum realm" (Penrose). Fortunately that's not how science works. | | |
| ▲ | dingaling911 an hour ago | parent | next [-] | | I understand what happened. You took my claim literally. Like DNA translates by a simple function to random initialization. I was saying that all else being equal, DNA will influence the way the brain initializes. Of course, many factors will influence it. | |
| ▲ | dingaling911 3 hours ago | parent | prev | next [-] | | Science starts with a hypothesis. If DNA has everything encoded, then it stands to reason that it influences the initial state of the brain. | | |
| ▲ | wernam 3 hours ago | parent [-] | | > If DNA has everything encoded It is not. There are many biological examples of random processes generating patterns and weights, such tiger stripes, fingerprints, etc. DNA only encodes the reaction, but the pattern itself is seeded by initial position of random molecules, not the DNA. > that it influences the initial state of the brain. You've gone from "DNA is probably the seed for the random initialization", as in the deterministic seed for a PRNG, to "influences". Those are 2 different claims. Most scientists will agree that DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc. Most scientists will disagree that it is "probable" that DNA encodes/seeds the weights of each and every synapse. | | |
| ▲ | alembic_fumes 3 hours ago | parent [-] | | I think you are mostly debating against a strawman of your own construction here. The person you're responding to never claimed that "seeding the random initialization" means literally the same thing as seeding a pseudo-random number generator, and the second message was likely intended as a clarification after being misunderstood. So it's not "2 different claims" contradicting; it's a single underlying claim being expressed through a lossy medium in an increasing level of detail. My own reading of the poster's intent aligns with this: "DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc." | | |
| ▲ | wernam 2 hours ago | parent [-] | | The grandparent comment is: "humans aren’t born with randomly initialised neural weights", with commenters throwing around harebrained hypotheses about those weights. Most biologists will fully disagree with the whole premise, and support the reverse claim that we are born with random weights, we learn to walk/speak English/etc after we're born. Not from weights seeded from our DNA. |
|
|
| |
| ▲ | spwa4 2 hours ago | parent | prev [-] | | Except Penrose is like Chomsky. They have past accomplishments, but nothing recent and everybody keeps bringing them up because they constantly comment on stuff so far outside of their area of expertise it's not funny anymore, it's just sad. At least Penrose had the good manners to make relevant contributions (meaning Chomsky effectively managed to steer his field in entirely the wrong direction, take everybody along with nothing to show for it. Nobody uses his contributions today in study. Yes he's celebrated ... but why? He's like a prime minister that presided over a country causing an economic crisis, throwing everyone into misery. I guess those are exactly the kind of politicians celebrated. Bush is celebrated while he destroyed half the economy, Clinton is forgotten (well, except for that one thing) despite being the best president of the last 50+ years economically) | | |
| ▲ | hardbass 5 minutes ago | parent [-] | | Most scientists think Penrose's theory of microtubules is wrong, however he is still far better than many consciousness theorists in giving an actual testable criterion. |
|
|
|
|
| ▲ | zardo 3 hours ago | parent | prev [-] |
| Tokens being passed on would be what, you close your eyes and experience your ancestors lives? |
| |
| ▲ | horsawlarway 3 hours ago | parent [-] | | No. More like, your ancestors that have a fight or flight response survive more often and have passed that down. Or that infants with a rooting response survive much more often. Basically, weights encoded biologically from hereditary behavior. Instincts. Now, the real conundrum is that if our DNA can really only encode ~12gb then how are we passing this information down? |
|
