One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones.

The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds.

The takeaway from this is that these approaches will always solve for specificity, but it's a much harder task to steer the model into making general solutions. So if you're an inference provider for some specific model shape, fantastic, go for it. If you are a maintainer of a open-source library, this is not useful.

▲

musebox35 an hour ago | parent | next [-]

This is one of the dilemmas that I am trying to wrap my head around. I love optimizing software pipelines, which often boils down to figuring out the operational constraints that the compiler and the generic libraries can’t assume. Then I exploit these to squeeze out performance. But in a world I can start from scratch and code a domain specific solution from line zero in a matter of hours/days, I do not need general libraries as much as I used to. On one hand the code won’t be as well tested as a good general library. On the other hand, it also won’t have a plethora of library bugs that are there because the code is generic and opaque. One counter argument is that things are never static and you can’t have specific code for too long. A counter to that is that you can then change the code to be specific to the new reality at very low cost. This is the mental loop I ride constantly. Disclaimer: My circumstances are definitely not general, I am not writing code that is truly large scale.

	▲	augment_me 11 minutes ago \| parent [-]
		Many open-source communities are wrestling with similar questions. In many ways the generic frameworks end up being reference implementations for the specific solutions to copy and hill-climb towards, and it's as you say unclear what the point is to maintain generic things beyond that. What I feel is lacking with the solutions that have for example agent-generated and tuned GPU kernels is that the use-cases for them are unclear. If you are a researcher on second-order optimizers, you probably want to be able to handle variable input shapes to experiment, you also might want something readable to understand intermediate steps and perhaps build on that. If you are a neolab running massive training runs for 80% of your VC funding, you need to know that every line in your training code is bitwise identical to the theory/reference because a divergent run from some LLM-generated numerical bug will set you back or bankrupt you, so you can't just plop in a random kernel even if it promises good performance. So who is the agentically-looped end result for? Except for Openai and Anthropic of course who sold the tools.

▲

josu an hour ago | parent | prev | next [-]

Overfitting to the input is part of the meta in this type of challenges.

The goal is not to create good, general or maintainable code. The only goal is to produce the fastest code.

	▲	piker 25 minutes ago \| parent \| next [-]
		Nonetheless gives appropriate context to the speed up for folks like myself who were just skimming.
	▲	dejavucoder an hour ago \| parent \| prev [-]
		fair argument

▲

dejavucoder an hour ago | parent | prev [-]

this is true. in one of the later problems (cholesky decomposition), the organizer ran the submissions on a tiny training run to validate... and also provided code for same for our reference. most of the top solutions hit 4/8 or so. not very numerically stable.

i found out that as i learnt more domain wise, i was (obviously) able to steer better. doing a re-write can also remove lots of slop and context rot (and subsequently make it easier for both human and LLM to make solution more numerically stable, less reward hackish)