Why is Pi not in the benchmarks? Deepseek beats Strands and its built on Pi so that’s all I needed to know.

cobolcomesback an hour ago | parent | next [-]

Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.

	▲	theturtletalks 35 minutes ago \| parent [-]
		I figured it was lack of MCP support. I would just include a note why it was excluded. Have you checked out FX? It seems to be like Pi but will more things out of the box like MCP.

▲

leodavi 2 hours ago | parent | prev | next [-]

Deepseek harness is not actually built on Pi harness. It's an independent project.

	▲	theturtletalks an hour ago \| parent [-]
		This is true, but the team said on Twitter that they were heavily inspired by Pi and this whole team used it extensively: https://x.com/tianyi/status/2088306143772946499

▲

techscruggs 2 hours ago | parent | prev | next [-]

Deepseek was cheaper, but also less accurate. "Beats" isn't a fair assessment.

	▲	theturtletalks 2 hours ago \| parent [-]
		That’s fair, but Strands is advertising their harness needing way less tokens which does relate to cost. In that same vein, Pi is less bloated then Deepseek and Oh My Pi, which are built on top of Pi. Isn’t it dubious to leave it out?

▲

jsw97 an hour ago | parent | prev [-]

Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.

	▲	strandstan an hour ago \| parent [-]
		hi Albert from the Strands team. We can look into a Pi run. Our researcher is working on a deep dive for our benchmarks, so we got resources to test against other harnesses