| ▲ | mikeshi42 a day ago |
| > Sharing matrices substantially reduces the weights that must be stored and moved while keeping the expert count unchanged. Each group of 64 experts shares two large matrices, while each expert retains a small set of parameters that captures its differences. 145B parameters with only peak 29GB memory usage - really impressive, truly something you can actually run on a decent spec MBP |
|
| ▲ | Aurornis a day ago | parent | next [-] |
| Their edge0-strata engine is just a fork of the strata project, which has been doing this for a long time: https://github.com/Niko1221/Strata They're not hiding that fact once you get to the repo. I mean it's called edge0-strata and the README says it's a fork of strata. It's weird to see this marketing web page imply that it was all new and novel work that they did, though. |
| |
| ▲ | dust42 19 hours ago | parent [-] | | And their model is just derived from Qwen3.8-next-flash which you can see when going to their HF page. Not much details to be found about the company apart from the founder who "was named in MIT Technology Review's Innovators Under 35". Not much innovation here though it seems. More like a marketing stunt to collect VC. Qwen3.8-next-flash is impressive though, especially with Strata or Basalt as inference engines. On a 5090 laptop with 64GB RAM one can run NVFP4 quants at 120-140t/s decode and 3000-4000t/s prefill. On Mac it is much slower and the Kapitsa engine probably the best bet. Despite the claim of 29GB RAM for Edge0, in order to be fast you need to have at least a 128GB Mac. | | |
|
|
| ▲ | wuschel a day ago | parent | prev | next [-] |
| I am not in the material, so perhaps someone else can comment on the quality of the outcome and technique? After skimming a page I am not really sure that the whole model fits inside 29 GB RAM. They seem to use some interesting techniques. ModelTokenMemoryUsage [Bytes] = Parameters * (BitsPerParameter/8) For a model with 145B parameters, this means: Precision GB GiB
2-bit 36.25 33.76
3-bit 54.38 50.64
4-bit 72.50 67.52
6-bit 108.75 101.28
8-bit 145.00 135.04
16-bit 290.00 270.08
32-bit 580.00 540.17
You have roughly a factor 8 in memory usage from 32 bit down to 4 bit precision. That is quite some room already, but not new.It seems that they radically reduce the model information required per token via the sparse expert approach - 145B->10B parameters, probably at lower precision - and then use some other approaches I do not understand at that point to make effective use of loaded information. Did anyone try the model? |
| |
| ▲ | daemonologist a day ago | parent | next [-] | | In the "Under the Hood of Edge0" section it mentions that it needs another 40 GB of system memory for experts plus 48 GB of SSD storage for vocabulary (engrams?). The 29 GB headline seems to just be a flat lie (/the LLM writing this article getting confused about what memory usage it's reading). | | |
| ▲ | Nursie a day ago | parent | next [-] | | Oh that’s annoying, from the hype I was hoping my 48GB MBP would be able to run this happily, but if we’re talking about -70GB combined RAM footprint then that’s not going to cut it. Still very interesting though. | | |
| ▲ | niek_pas a day ago | parent [-] | | The image under “Competitive capability with a smaller memory footprint” explicitly claims: > Edge0-145B scores above comparable models with less than half the memory (29 GB vs ~70 GB) and runs on a 48GB laptop So either they’re lying or confused or I’m confused. |
| |
| ▲ | nghiemphucminh7 a day ago | parent | prev [-] | | [flagged] |
| |
| ▲ | Tade0 21 hours ago | parent | prev | next [-] | | Most likely a huge part of the magic is keeping the ngram table on disk, as only one row from it is read per token, so there's next to no performance hit. Qwen 3.8 Flash Next utilizes this, with 40% of its parameters being offloaded to disk, making the effective memory requirements much lower. My daily driver is this: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC... It takes things a step further by simply chopping off that half of experts which isn't (or rarely is) activated in coding tasks. End result is a somewhat competent [0] coding model which runs on a 64GB machine. [0] it still occasionally does weird stuff like replacing an entire file with just one line from its answer when you ask it a question if you don't explicitly tell it to not change anything or loop itself if the KV Cache is quantised at below Q8_0. | |
| ▲ | mapontosevenths 17 hours ago | parent | prev [-] | | I tried it - it doesn't work with my configuration and setup on a DGX Spark. I won't post my prompt here (I don't want it to get benchmaxxed), but I use a python programming question that used to be a common interview question 10-20 years ago to test models. It spent 8,000+ tokens at default thinking and never produced a coherent answer. This is essentially a very clever concept that isn't really ready yet. Or perhaps the custom inference engine just doesn't play nice on my hardware yet. Either way, it's not something I'll be able to actually use in it's current state. |
|
|
| ▲ | theroadnotbacon a day ago | parent | prev | next [-] |
| The blog post says: > For Edge0-145B, day-one support covers macOS, Windows, and Linux, including Apple M-series chips and NVIDIA platforms such as RTX Spark, RTX 4090, and RTX 5090. The release includes one-click deployment tools and hardware guides. But... there's no guidance on how to run on a mac at all. Weird. |
| |
|
| ▲ | a day ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | origini 18 hours ago | parent | prev | next [-] |
| Don't forget the memory requirements for context. |
|
| ▲ | embedding-shape 17 hours ago | parent | prev [-] |
| > 145B parameters with only peak 29GB memory usage - really impressive They're doing offloading, the full model doesn't live in VRAM at all. They're expecting you to do ~30GB VRAM + the rest on system RAM. |
