| ▲ | LoveMistral 4 hours ago |
| Same. Mistral 7b has been more than I ever needed for text for years now. Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid. Mistral 7b can do anything, and it’s basically instant even on an M3 |
|
| ▲ | frigidwalnut 4 hours ago | parent | next [-] |
| Sounds interesting. Can you give more details on your workflow and what tasks you use it for? |
| |
| ▲ | LoveMistral 3 hours ago | parent [-] | | Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work. Actually built a full invoicing product for that, using it too. I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC). Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :) | | |
|
|
| ▲ | Almondsetat 3 hours ago | parent | prev | next [-] |
| What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much. |
| |
| ▲ | LoveMistral 3 hours ago | parent [-] | | RAG is your friend (or any vector db). No model can vectorize an entire codebase in context. Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details. What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response. If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use. |
|
|
| ▲ | casper14 3 hours ago | parent | prev [-] |
| What are some limitations you have found with using a smaller model like that? |
