I tested with

  import needle

  @needle.tool
  def add(a: int, b: int):
      "Add two numbers."
      return a + b

  agent = needle.Needle(tools=[add])
  print(agent.run("calculate 1 + 1?")["reasoning"])

python main.py No calculator or math tool available.

conclusion: completly useless

▲ HenryNdubuaku 2 hours ago | parent [-]

Try the following tool description: "Calculate the sum of two numbers. Use for any arithmetic or math question." instead of "Add two numbers." Let me know how it goes, thanks!

▲ HenryNdubuaku 2 hours ago | parent | next [-]

It does better with clearer tool description, but we are taking note of these complaints for future improvements.

▲ hathym 2 hours ago | parent | prev [-]

works better that way, thanks :)

▲ hathym 2 hours ago | parent | next [-]

but still struggle when changing the quesion:

  import needle

  @needle.tool
  def add(a: int, b: int):
      "Calculate the sum of two numbers. Use for any arithmetic or math question."
      return a + b

  agent = needle.Needle(tools=[add])
  print(agent.run("what is 5 + 7?")["reasoning"])

>> No calculator or math tool available. Cannot compute numbers.

	▲	HenryNdubuaku 2 hours ago \| parent [-]
		Ah, another failure point on our end! So a simple "5 + 7" and "add 5 and 7" works. But to handle ambiguity, the python package ships pipelines to synthesize augmentations and fine-tune on your samples for robustness. Just run "needle playground" and use the UI. Apologies.

▲ HenryNdubuaku 2 hours ago | parent | prev [-]

Thanks, we shall improve this for the next release.