Show HN: Flide: Make Jev-like decision models up to 9.5x faster

Decision models need their own inference engine.

So we developed Flide, a high-performance and exact inference engine for Jev-like decision models.

Flide achieves up to a 9.5x speedup over vLLM and SGLang.

It now supports 30+ popular local Jev-like decision models.

Start to make your decision immediately:

pip install flide

▲

yossuf2000 27 minutes ago | parent | next [-]

do i need to host the model to do this ? if yes what model can i host in 3050 card

▲

jarym an hour ago | parent | prev | next [-]

I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!

▲

unikcc 5 hours ago | parent | prev [-]

Hi HN! I'm one of the Flide team.

We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.

Here is the source code: https://github.com/OmniJev/Flide

Look forward to hearing your feedback and ideas!

▲

rytill 2 hours ago | parent [-]

Under what specific circumstances is it actually 9.5x faster?

Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?

How did you achieve the biggest speed up?

Are there any tradeoffs? “While keeping exactness” implies not.

Are there any specifics about how one must structure their context to take full advantage of it?

	▲	tancop an hour ago \| parent [-]
		I looked at their paper and the biggest gains come when you have many questions from one context, which is the way these models are supposed to be used. It's around 1.5-3x faster than vLLM with single questions. That's the worst case where you can't reuse anything. The way I understand it most of their speedup comes from caching intermediate results. They freeze the state after processing context then branch each question off that point. It means they can process questions in parallel after the context stage is finished, and don't waste time doing the same work N times.