"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

https://news.ycombinator.com/newsguidelines.html

As a hacker, I kinda like naom's code. I was had to implement a TC MoE kernel, and stumbled upon his code from [tensor2tensor](https://github.com/tensorflow/tensor2tensor/blob/master/tens...) and i think "alchemy" is justified. Dude writes some beautiful kernels.

He also saw LLM would replace search before anyone else, and that is something to look at the Lamda or GPT-1's output and think: yeah this will answer all of our questions one day.

The architecture was Shazeer's, but the rough idea came from Jakob Uszkoreit who initiated the project.

Uszkoreit wanted to build a more efficient/scalable language/seq2seq model that could take advantage of GPU parallelism (replacing RNNs which were the main approach to sequence modelling at that time).

Uszkoreit's insight was that although language appears sequential, it is in fact really part parallel part hierarchical, as can be seen by linguist's sentence parse trees where at each level there is parallelism/independence between the branches of the tree, with them getting combined at the next level up. This is what gave rise to the idea of a model that consisted of a stack of of parallel processing layers (transformer layers). I believe that attention was also part of the plan from day one, as this had already been proven to be valuable (Bahdanau) with RNN seq2seq modelling.

So, this is what Uszkoreit wanted to build, but by his own account he failed to come up with an implementation that matched or outperformed the prevailing RNN approach that he wanted to replace. At this point, Uszkoreit mentioned the idea to Shazeer, who got on board and eventually arrived at a performant architecture which was then pared back by an ablation process resulting in the initial encoder-decoder Transformer architecture. Shazeer later came up with the mixture-of-experts architecture, and also other optimizations after he left to found character.ai

Source for this? The notion of attention dates to a content-addressable lookup during sequence alignment (as well as, concurrently, memory lookups in neural Turing machines). Attention had been used in other models, like GRUs and LSTMs with attention. The Vaswani et. al. paper did not introduce attention, just removed everything _but_ attention (and FFW) from the network. Are you claiming the "critical idea" of removing the GRU and LSTM parts and just keeping attention was "truly" Noam's?

I don't know we can just say things now. Ah we're on the internet

Is this a generally well known thing?

Even more important, I wonder what it says about HBW...

Yeah my current feeling is that once I had double digit millions earning further money would be pretty meaningless to me, and the difference between 'large salary' and 'even larger salary' would be even more meaningless, but who knows maybe it really would change me. I kind of assume people like this are primarily chasing the most interesting/impactful work though.

It gets to the point where what you do is the main question while payment is barely a minor concern way earlier than that point, at least in my experience. You don't need to be in the top AI research tier for that.

OpenAI pays for the earn out he would’ve otherwise received at Google + a new comp package. Made up numbers, if Google still owed him $10M for lasting the full two years, OpenAI can just pay him market rate +$10M.

Right?! Unless you think this move is going to generate general excitement in our lives, it's just another rich guy moving from one high paying job to another.

Maybe he figured out a good way to short AI companies?

Oui!

The Netflix documentary will reveal he was secretly working for Sam Altman the whole time... (Cue diabolical VC-backed evil laugh.)

Google lost three critical years chasing AGI, and got acquired by SpaceX, now a Dyson Sphere startup whose pitch deck is just: "What if we put a paywall around the Sun?"

That twitter story isn't anything unique to OpenAI or Google, it's just classic "big public corp vs private startup" culture. Once you have to worry about the SEC, shareholders, antitrust, regulations, lawsuits, etc. it's very, very difficult to avoid turning into "big corp" culture.

Sama, and any other founder, will always have a difficult fight against bureaucracy, and once you let a little bit in, the bureaucracy's sole purpose becomes to grow itself.

He is close or already a billionaire, not sure much more money will be do much heavy-lifting

I think real life has far eclipsed the absurdity of the original show. They might have a hard time competing with just the news now days.

Three comma guy would now be four comma guy

I loved that show. The love that went into it really shows.

Sadly the gap between reality and satire has shrunk.

But yes. I also wish that show would come back.

Noam shazeer would be google head dreamer

Gilfoyle was really ahead of the times with Son of Anton.

Your dream may be only a prompt away.

What does Zionist mean when Israel has existed as a Jewish state for 78 years? I'm genuinely asking because the way the word is used doesn't make sense to me. There aren't similar terms for other countries to just stay the same, like for China to keep being run by the CCP. Every other country is assumed to have ontological inertia except for Israel.

https://x.com/karpathy/status/2056753169888334312?lang=en

I think the 'models have no moat' thing is overblown. Only like 3-4 companies in the entire world have cutting edge models, that means there is some kind of moat...

yeah, sure, look at anthropic revenue, what is it if not the moat? you can argue for how long but for them good model = the fastest growing company ever.

I feel like the models have no moat paradigm died when a single model expanded past the memory of single GPU slices. The moat is hosting the model. Even paying a server host to run a rack of GPUs has immense upstart cost, and then you're still struggling to compete on the add-ons of the things on top of the model (prompts, validation loops, etc). You can only throw so much money at a problem.

And they've had some initial success with TPUs which could be a major differentiator in the future.

> models have no moat

Possibly true. Any smart innovations developed by one organization will be smuggled into others.

Training, inferring, and data collection, infrastructures are definitely moats. High-volume usage feedback is also hard to come by for new entrants.

I don't think you're honestly accounting for the engineering behind the progress models are making. If it was just a matter of compute on hand and iterating, Meta would be neck and neck with Ant, OAI, and Google, but clearly you've gotta have more.

Noam has a deep expertise in these systems at every level, both algorithmically and at production scale, and knows how to leverage things at different levels.

It's not like Google won't have anyone else that can do what he does, but at the same time, it's an implicit criticism of Google's culture, operations, development, and overall AI program. Shazeer is well past the point where the paycheck is the deciding factor, although I'm certain he is very well paid. Having the freedom to innovate and build free from the corporate fuckery of Google and Facebook is probably more valuable than the pay raise he got with the move, and OAI has the advantage of not having to cope with decades of corporate cruft and inertia. They'll get there - all corporations do - but they're relatively young enough to still be nimble.

Their options are probably insane sunk cost, hard to steal an engineer who has Xm in potential gains if they choose to stay.

Love the choice of words by Noam- exceptional team for OpenAI, amazing team for GDM.

I wonder if the ideological censorship described in your link is part of why Noam decided to leave.

How do I ̷g̷a̷m̷b̷l̷e̷ sports bet on this

I feel like there was a scene in Silicon Valley about a developer fantasy league.

Idk, football players actually make a bunch of people happy and entertained. 80% of the United States wishes this tech never existed.

What they're working on is just making peoples jobs, skills obsolete and trying to invent machines that will concentrate the worlds wealth into the hands of the people who own those machines.

This situation is kind of like backend NIL value. His value to OAI isn't just the work he'll do "on the playing field", it's the perceptual value of "OAI just hired the guy Google paid >$2B to get back" right before their IPO.

OpenAI hasn't released open weights since GPT-OSS-20/120B. Google has the Gemma line.

I wouldn't expect OpenAI to start releasing open weight competitive models again, but I could be wrong.

Yah, I guess Cali doesn't allow non-competes or something like that.

LOL.

And threw in a sweetener when he seemed hesitant: he can say whatever he likes about trans people in the workplace and not worry about being PC.