> I think the more productive way for frontier models to interact with a unity project is to directly manipulate the raw scene files.

I disagree. Even moreso than webdev, gamedev benefits from a fast and reliable loop, especially one that can capture realtime state and screenshots. It's the difference between having a new well tested mechanic in your game next morning and giving up after a week of debugging.

Agents will work around slow iteration loops perfectly fine given enough time, but when you're shipping 20 core things in your game each day, the agent's test harness becomes the bottleneck.

IMO it's worth it to spend a day or two to make sure your test loop is tight, it will pay dividends. Not saying Unity is great at playing ball here though. Hopefully their new effort helps.

▲

bob1029 6 hours ago | parent [-]

Letting the agent take screenshots of the scene is not effective. A human needs to be in that loop.

You can get primitive layouts working if you feed it orthographic projections from the 6 faces, but it falls apart once you need to start looking at things in perspective. It's also absolutely ass at interpreting scene illumination.

We need models trained on 3d scene transforms (aka world models). Passing raster images into the model is a really inefficient method of sending this logical information.

	▲	DonHopkins 3 hours ago \| parent [-]
		I wouldn't be so sure about that. Have you actually tried? You might be surprised. Screenshots are a hell of a big baby to be throwing out with the bathwater. The structured data this cli returns is great, but screen snapshots can be extremely useful too, both together. Editing raw yaml files can be tricky, so you do want a formal editing api you can go through, that is extensible for your own component property getters and setters and apis and editors, that correctly maintains all the constraints and dependencies, instead of just letting the LLM hot dog it with your raw yaml files by peeking and poking.