You may benefit from an embedding approach for semantic search. Not sure what an LLM would give you on top of that.

> You will probably want a search engine though.

The search engine is indeed the last missing component from a sovereign stack. But I think this could be solved locally with little cost. Instead of indexing content on the web we should be indexing sources themselves - where to look for X? - like forums, blogs, docs, feeds, and specialized search engines. We could collectively amass millions of these search stubs that can be used by local models to go and fetch fresh information from the source directly. This means separating the routing layer from the information layer, we don't need to keep information cached from the whole internet locally. The search stubs could fit in a few GB about same size with the local LLM. The cool thing is that sources change much slower than information itself, so the search stub database could be refreshed at a slower pace. We could combine a few million generic stubs with a few hundred personal stubs generated from our own activities. It is trivial to generate these stubs by piggy backing on frontier models.

grep works well!

I feel this is going into increasingly-unlikely mixes of constraints and needs in order to try to keep a "wouldn't it be cool if" hypothetical-tool dream alive. [0]

But OK, let's assume that: The power is out, but you have a generator with so much fuel you can run a desktop just fine; Your neighborhood will somehow make a mesh network; Your neighbors need some already stored information and the best solution for that is texting a chatbot rather than a survival/emergency handbook or Wikipedia; Your mesh-network will also be good enough to match the time-sensitivity of the questions.

Under those assumption, which of these sounds better?

1. Buying an "LLM-in-a-box for emergency supply kits", which you deploy so that your neighbors can ask questions (text over the mesh) of the offline chatbot.

2. Buying a satellite internet transciever for your emergency supply kit, so that your neighbors can ask questions of a much better chatbot and communicate with human experts, their worried relatives, and coordinate with rescue/relief efforts...

[0] https://xkcd.com/2128/

There is no rebuilding society because of energy. All of the oil and coal that could be extracted by a civilization that only has wood for fuel has been used. If we go back to the Stone Age, we aren’t returning to the present. If we’re lucky, we will get to the Middle Ages.

No doubt there are a hundred more direct things one could do. On the other hand this would include instructions for all those things and could advise on the building! I think it would be a nice thing to have in the world.

> They want to ask the iOS Foundation model (frontier on device intelligence for something small)

This is a bit of a straw man, TBH.

For one thing, "LLM-in-a-box" doesn't necesssarily imply a device as small as a phone.

For another, you'd need to convince people that the iOS Foundation model is the "frontier" of LLMs that run on phones when it is really not. AFAIK it is noticeably outperformed by the Gemma 4 E2B model and certainly the E4B.

https://blog.google/innovation-and-ai/technology/developers-...

Here is a common-or-garden youtube video that includes a demonstration of how much better the E2B is:

https://www.youtube.com/watch?v=sTxyBUbdZcA

Whether this idea (LLMs for emergency/survival scenarios) has value, I don't know, so I am not offering an opinion, but you should approach it with a good faith argument.

I am an LLM cynic but I suppose if I was to be without connectivity but with power for a while, a device with the Gemma 4 E2B or E4B model on it might be helpful or interesting to have. If such a device had the 12B QAT model on it, that really would cross the line to utility. Not sure it has value in the OP's scenario, still.