You might be able to do it with ibis. Don't know about the performance though

Thank you, I'll look into this. Yes performance is the main driver when some data frames have millions of rows.