I am not in the material, so perhaps someone else can comment on the quality of the outcome and technique? After skimming a page I am not really sure that the whole model fits inside 29 GB RAM. They seem to use some interesting techniques.

ModelTokenMemoryUsage [Bytes] = Parameters * (BitsPerParameter/8)

For a model with 145B parameters, this means:

    Precision       GB       GiB
    2-bit         36.25     33.76
    3-bit         54.38     50.64
    4-bit         72.50     67.52
    6-bit        108.75    101.28
    8-bit        145.00    135.04
    16-bit       290.00    270.08
    32-bit       580.00    540.17

You have roughly a factor 8 in memory usage from 32 bit down to 4 bit precision. That is quite some room already, but not new.

It seems that they radically reduce the model information required per token via the sparse expert approach - 145B->10B parameters, probably at lower precision - and then use some other approaches I do not understand at that point to make effective use of loaded information.

Did anyone try the model?

▲ daemonologist a day ago | parent | next [-]

In the "Under the Hood of Edge0" section it mentions that it needs another 40 GB of system memory for experts plus 48 GB of SSD storage for vocabulary (engrams?). The 29 GB headline seems to just be a flat lie (/the LLM writing this article getting confused about what memory usage it's reading).

▲

Nursie a day ago | parent | next [-]

Oh that’s annoying, from the hype I was hoping my 48GB MBP would be able to run this happily, but if we’re talking about -70GB combined RAM footprint then that’s not going to cut it.

Still very interesting though.

	▲	niek_pas a day ago \| parent [-]
		The image under “Competitive capability with a smaller memory footprint” explicitly claims: > Edge0-145B scores above comparable models with less than half the memory (29 GB vs ~70 GB) and runs on a 48GB laptop So either they’re lying or confused or I’m confused.

▲

nghiemphucminh7 a day ago | parent | prev [-]

[flagged]

▲ Tade0 21 hours ago | parent | prev | next [-]

Most likely a huge part of the magic is keeping the ngram table on disk, as only one row from it is read per token, so there's next to no performance hit.

Qwen 3.8 Flash Next utilizes this, with 40% of its parameters being offloaded to disk, making the effective memory requirements much lower.

My daily driver is this:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

It takes things a step further by simply chopping off that half of experts which isn't (or rarely is) activated in coding tasks.

End result is a somewhat competent [0] coding model which runs on a 64GB machine.

[0] it still occasionally does weird stuff like replacing an entire file with just one line from its answer when you ask it a question if you don't explicitly tell it to not change anything or loop itself if the KV Cache is quantised at below Q8_0.

▲ mapontosevenths 17 hours ago | parent | prev [-]

I tried it - it doesn't work with my configuration and setup on a DGX Spark.

I won't post my prompt here (I don't want it to get benchmaxxed), but I use a python programming question that used to be a common interview question 10-20 years ago to test models. It spent 8,000+ tokens at default thinking and never produced a coherent answer.

This is essentially a very clever concept that isn't really ready yet. Or perhaps the custom inference engine just doesn't play nice on my hardware yet. Either way, it's not something I'll be able to actually use in it's current state.