This is a solid improvement over Needle 2, which I tried using for a tool-calling interface to a Runescape database site. Unfortunately it's still not quite capable enough for my target compared to FunctionGemma.

   Model                            Correct tool shape    Exact arguments
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━
   FunctionGemma fine-tune, BF16       209/230 (90.9%)    196/230 (85.2%)
  ───────────────────────────────  ────────────────────  ─────────────────
   Needle 3 fine-tuned W4A8             74/230 (32.2%)     47/230 (20.4%)
  ───────────────────────────────  ────────────────────  ─────────────────
   Needle 2 fine-tuned W4               59/230 (25.7%)     43/230 (18.7%)

Thanks for testing Needle out! I'd be very interested in hearing more about the finetuning setup to see how we can make both the library's finetuning setup and the model better.