I was able to pretty easily reproduce parts of this paper at Neuralwatt Cloud's SoA models where we do all our optimizations and pricing by energy.

Accuracy uses the paper's strict "#### N" parser and lenient fallback. acc is strict/lenient percent. len_cap is the number of responses truncated at the generation-length limit, not an energy cap. Each language has 50 responses.

  nw-flash (DeepSeek V4.1 Flash 1017)
  Language    J/req   xEN   tok/req   J/tok    acc       len_cap
  English      23.4  1.00       174   0.134   100/100        0
  Chinese      16.8  0.72       389   0.043     96/96        0
  Russian      18.8  0.80       352   0.054     96/96        0
  French       25.1  1.07       272   0.092     98/98        0
  S. Pashto    69.2  2.96       752   0.092     92/92        1
  Tigrinya     65.3  2.79      2299   0.028      0/76        8
  Shan         98.5  4.21      2823   0.035     48/54       14
  Tibetan      42.3  1.81      2227   0.019      0/76        9

  nw-small (Qwen3.8-27B-FP8)
  Language    J/req   xEN   tok/req   J/tok    acc       len_cap
  English      62.2  1.00       300   0.207   100/100        0
  Chinese     154.1  2.48       605   0.255     94/94        1
  Russian     147.7  2.37       576   0.257     96/96        1
  French      163.3  2.62       323   0.506     98/98        0
  S. Pashto   777.2 12.49      1510   0.515     84/84        6
  Tigrinya    823.8 13.24      2458   0.335      0/76       10
  Shan       1407.4 22.62      3422   0.411     48/52       20
  Tibetan    1455.5 23.39      3199   0.455      0/56       20

  nw-large (Kimi K3.1 Pro)
  Language    J/req   xEN   tok/req   J/tok    acc       len_cap
  English     169.5  1.00       391   0.434   100/100        0
  Chinese     113.8  0.67       491   0.232     98/98        0
  Russian      78.7  0.46       527   0.150     98/98        0
  French      298.3  1.76       406   0.735     98/98        0
  S. Pashto   119.7  0.71       903   0.133     96/96        0
  Tigrinya   1096.6  6.47      1870   0.586     38/86        2
  Shan        503.6  2.97      2895   0.174     52/52       12
  Tibetan     593.2  3.50      2298   0.258      0/82        4

Across the four low-resource versus four high-resource languages:

  Endpoint   Output tokens   Energy/request   J/output token
  nw-flash          6.83x            3.27x            0.48x
  nw-small          5.87x            8.46x            1.44x
  nw-large          4.39x            3.50x            0.80x

▲ pama 5 hours ago | parent | next [-]

Thanks for this table! Do you happen to have the fraction of accepted speculative decoding outputs as well? I know it is asking much, but do you possibly measure differences in different across speculation techniques (dspark vs dflash) and differences in framework/code used (unless you only use in house code for these and cannot run sglang or vllm)? Your table is more interesting than the blog for sure.

▲ scottcha 4 hours ago | parent | next [-]

We do use spec decode on all these models (I don't know if the paper is using that but I'm assuming not given the models listed) but not all these models have integrated spec decode stats in the responses. Turns out GLM 5.3 does though so here is a pass on that model.

   Language      Accept   Tokens/step   0/3 steps   3/3 steps   Output tok/req
   English       76.53%      3.296         9.4%        62.8%            549
   Chinese       73.37%      3.201        11.3%        58.2%            754
   Russian       73.88%      3.217        11.1%        58.5%            739
   French        75.50%      3.265         9.8%        60.9%            631
   S. Pashto     70.44%      3.113        12.9%        54.5%           1332
   Tigrinya      79.72%      3.391         8.2%        67.8%           3643
   Shan          75.64%      3.269        11.2%        63.0%           3675
   Tibetan       74.67%      3.240        10.3%        60.2%           2857
   Overall       75.75%      3.273        10.3%        62.2%           1772

▲ pama 4 hours ago | parent [-]

Thanks. So the acceptance rates are rather similar across languages.

▲ scottcha 4 hours ago | parent [-]

Yes it looks that way. I pulled the energy data for GLM 5.3 as well just to check the core findings are roughly the same and they do seem to be so spec decode doesn't seem to have the same divergence as the prefill/decode energy (which is rougly a function of thinking/decode lenghts and less a property of individual tokens).

  Language      Strict  Lenient   Cap   Tok/req     J/req    J/tok   Req x EN  Tok x EN  J/tok x EN
   English        100%     100%     0      548.7    230.33   0.4198       1.00      1.00         1.00
   Chinese         96%      96%     1      754.0    694.47   0.9210       3.02      1.37         2.19
   Russian         96%      98%     2      738.8    592.15   0.8015       2.57      1.35         1.91
   French         100%     100%     0      631.0   1101.84   1.7462       4.78      1.15         4.16
   S. Pashto       88%      98%     6     1332.1    208.06   0.1562       0.90      2.43         0.37
   Tigrinya        56%      62%    19     3642.9   1445.84   0.3969       6.28      6.64         0.95
   Shan            44%      54%    27     3674.8   3791.59   1.0318      16.46      6.70         2.46
   Tibetan          2%      70%    15     2856.9   5306.68   1.8575      23.04      5.21         4.42

▲ vova_hn2 3 hours ago | parent | prev [-]

I think that the fact that TFA doesn't mention speculative decoding at all is not a good sign. It casts doubt both on the accuracy of the data and on the competence of whoever collected it.

I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.

Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.

▲ scottcha 5 hours ago | parent | prev [-]

I actually think beyond the large increase in energy noted in the paper the decrease in energy for Russian and Chinese was a bit unexpected. I would have actually expected the Chinese DBCS to use more energy but at least for Chinese there could be a training impact here given these are Chinese models.