I agree- I'm currently trying to learn how I can embed a fine tuned tiny model into my c++ game so it can provide a narrative in prose of certain game-event logs. It needs to be as tiny as possible so it doesn't take resources away from the running game.

▲

lelanthran 2 hours ago | parent | next [-]

> I agree- I'm currently trying to learn how I can embed a fine tuned tiny model into my c++ game so it can provide a narrative in prose of certain game-event logs.

Unless your game states have combinatoral exlosion, would it not be better to generate all of that pre-build? If templated you can generate a few hundreds of thousands of templates to use for any circumstance, then instantiate and stitch together those templates during the game runtime.

▲

yw3410 4 hours ago | parent | prev [-]

How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory?

	▲	lelanthran 2 hours ago \| parent [-]
		> How small a model are we talking? Don't even the smallest models which would work need gigabytes of memory? I dunno, for game prose I expect that a tiny highly quantized model would be sufficient (generating no more than a paragraph), so 300MB - 500MB maybe? Running on CPU not GPU is feasible too, I think.