new | show | ask | jobs Github

refulgentis 4 days ago

[flagged]

▲

HenryNdubuaku 4 days ago | parent | next [-]

Thanks for the comment, but:

1) The commit history goes back to April.

2) LlaMa.cpp licence is included in the Repo where necessary like Ollama, until it is deprecated.

3) Flutter isolates behave like servers, and Cactus codes use that.

▲

refulgentis 4 days ago | parent [-]

[flagged]

▲

HenryNdubuaku 4 days ago | parent [-]

We are following Ollama's design, but not verbatim due to apps being sandboxed.

Phones are resource-constrained, we saw significant battery overhead with in-process HTTP listeners so we stuck with simple stateful isolates in Flutter and exploring standalone server app others can talk to for React.

For model sharing with the current setup:

iOS - We are working towards writing the model into an App Group container, tricky but working around it.

Android - We are working towards prompting the user once for a SAF directory (e.g., /Download/llm_models), save the model there, then publish a ContentProvider URI for zero-copy reads.

We are already writing more mobile-friendly kernels and Tensors, but GGML/GGUF is widely supported, porting it is an easy way to get started and collect feedback, but we will completely move away from in < 2 months.

Anything else you would like to know?

▲

refulgentis 4 days ago | parent [-]

How does writing a model into an App Group container enable your framework to enable an app to enable a local LLM server that 3rd party apps can make calls to on iOS?[^1]

How does writing a model into a shared directory on Android enable a local LLM server that 3rd party apps can make calls to?[^2]

How does writing your own kernels get you off GGUF in 2 months? GGUF is a storage format. You use kernels to do things with the numbers you get from it.

I thought GGUF was an advantage? Now it's something you're basically done using?

I don't think you should continue this conversation. As easy it as it is to get your work out there, it's just as easy to build a record of stretching truth over and over again.

Best of luck, and I mean it. Just, memento mori: be honest and humble along the way. This is something you will look back on in a year and grimace.

[^1] App group containers only work between apps signed from the same Apple developer account. Additionally, that is shared storage, not a way to provide APIs to other apps.

[^2] SAF = Storage Access Framework, that is shared storage, not a way to provide APIs to other apps.

▲

HenryNdubuaku 4 days ago | parent [-]

[flagged]

	▲	jeffhuys 4 days ago \| parent \| next [-]
		The best way to go about this is realizing that there are more people reading this thread that make their own assumptions. Not staying professional and just answering the questions, and just doing "aight im outta here" when it gets a little bit harder is not a good look; it seems like you can't defend your own project. Just FYI.
	▲	refulgentis 4 days ago \| parent \| prev [-]
		[flagged]

▲

rshemet 4 days ago | parent | prev [-]

reminds me of

- "You are, undoubtedly, the worst pirate i have ever heard of" - "Ah, but you have heard of me"

Yes, we are indeed a young project. Not two weeks, but a couple of months. Welcome to AI, most projects are young :)

Yes, we are wrapping llama.cpp. For now. Ollama too began wrapping llama.cpp. That is the mission of open-source software - to enable the community to build on each others' progress.

We're enabling the first cross-platform in-app inference experience for GGUF models and we're soon shipping our own inference kernels fully optimized for mobile to speed up the performance. Stay tuned.

PS - we're up to good (source: trust us)