I do something related in my gpu library. After a few frames if the push constants don't change I compile the shaders in the background with them defined out by the preprocessor to reduce the size of the shader program (kind of like a branch predictor). I also store the entire pipeline in a graph data structure that I partition into segments that let me fuse and split kernels (though I hadn't implemented those optimizations yet). In my mind one issue with these GPU accelerated programs is that there isnt a runtime with the right level of information about the overall program to do compiler style optimizations, especially for complex programs.

(The library is called goldy, and until I spend some time on it the readme and docs are sadly LLM generated)

▲

bhargavkk 5 hours ago | parent [-]

Hey, first author here. You are correct when you say that GPU programs do not have the right amount of information to do compiler style optimizations, which is why its important to find the right abstraction level. Even compilers do not directly optimize assembly as assembly has little to no information about the original source code; they usually do it all on an IR that is carefully engineered to hold all the useful information needed to optimize programs.

	▲	mohamedkoubaa 5 hours ago \| parent [-]
		GPU programs as they are currently authored. It's because there isn't a runtime or DSL at the right conceptual level that is still practically useful against modern hardware and APIs. This is what I wanted to tackle with goldy. Goldy has the high level structure of the shader graph and exchanges with the outside world (host memory, surfaces, etc) which is turned into a backend specific graph IR (different backends have different rules for command list retention and other properties). But my library also embeds Slang and uses Slang IR to inject and introspect on the user written shaders. I can then at runtime turn it into CUDA graphs (slang can turn any shader into a CUDA kernel) but with the added benefit that the runtime can manipulate the kernels and graph as more information is available. For example if a host upload that usually is scheduled between kernels isn't done on a specific graph submission, the graph partitioner can fuse kernels and remove a fence. I plan to take advantage of the compiler and runtime introspection to implement some interesting features like shader coroutines without sacrificing GPU residency.