On the other hand, you could take a list of primitives from, say, the JS Canvas API, come up with a format that can encode all of them and use that as your protocol. Bam, with that you get one client for any game that uses JS Canvas.

That's exactly my approach! I'm sending canvas commands instead of pixels, which makes things faster

That said.. I don't think stadia could do that since it's not opinionated about the game engine. Unless they go really deep on the graphics card instructions instead, but then it becomes comparable to pixel rendering I guess