Ollama runs on Android just fine via Termux. I use it with 5GB models. They even recently added ollama package, there is no longer need to compile it from source code.

▲

rshemet 4 days ago | parent | next [-]

True - but Cactus is not just an app.

We are a dev toolkit to run LLMs cross-platform locally in any app you like.

▲

jadbox 4 days ago | parent | next [-]

How does it work? How does one model on the device get shared to many apps? Does each app have it's own inference sdk running or is there one inference engine shared to many apps (like ollama does). If it's the later, what's the communication protocol to the inference engine?

	▲	rshemet 4 days ago \| parent [-]
		Great question. Currently, each app is sandboxed - so each model file is downloaded inside each app's sandbox. We're working on enabling file sharing across multiple apps so you don't have to redownload the model. With respect to the inference SDK, yes you'll need to install the (react native/flutter) framework inside each app you're building. The SDK is very lightweight (our own iOS app is <30MB which includes the inference SDK and a ton of other stuff)

▲

pogue 4 days ago | parent | prev [-]

I would like to see it as an app, tbh! If I could run it as an APK with a nice GUI interface for picking different models to run, that would be a killer feature.

▲

rshemet 3 days ago | parent [-]

https://play.google.com/store/apps/details?id=com.rshemetsub...

	▲	pogue 3 days ago \| parent [-]
		Ah ha!

▲

v5v3 4 days ago | parent | prev [-]

Didn't know that. Thanks