commentators here are oddly obsessed with local serving imo, it's essentially never practical. it is okay to have to rent a GPU, but open weights are definitely good and important.

▲

nutjob2 4 hours ago | parent | next [-]

It's not odd, people don't want to be dependent and restricted by vendors, especially if they're running a business based on the tool.

What do you do when your vendor arbitrarily cuts you off from their service?

▲

nl 3 hours ago | parent | next [-]

You switch to one of the many, many other vendors serving the same open model?

	▲	Zetaphor 22 minutes ago \| parent [-]
		There can be quality differences across vendors for the same model due to things like quantization or configuration differences in their backend. By running locally you ensure you have consistency in addition to availability and privacy

▲

whimsicalism 3 hours ago | parent | prev [-]

i am not saying the desire to be uncoupled from token vendors is unreasonable, but you can rent cloud GPUs and run these models there. running on your own hardware is what seems a little fantastical at least for a reasonable TPS

	▲	pixelpoet an hour ago \| parent [-]
		I don't understand what is going on with people willing to give up their computing sovereignty. You should be able to own and run your own computation, permissionlessly as much as your electricity bill and reasonable usage goes. If you can't do it today, you should aim for it tomorrow. Stop giving infinite power to these rent-seeking ghouls! Be grateful that open models / open source and semi-affordable personal computing still exists, and support it. Pertinent example: imagine if two Strix Halo machines (2x128 GB) can run this model locally over fast ethernet. Wouldn't that be cool, compared to trying to get 256 GB of Nvidia-based VRAM?

▲

retr0rocket 4 hours ago | parent | prev [-]

[dead]