I used the word firehose, which is typically reserved for streaming data and is a whole other very interesting problem space! Big Data vs Fast Data.

I’m just noting for interest that firms are applying transformers and other networks at this streaming microstructure level, but specially trained for feature extraction. HRT + Nvidia have some nice videos about it

I will also note that it is insane how much better all the LLMs are at calling MCP tools after just a year, especially the local ones.

One of the reasons I like DuckDB is the scale flexibility. I started with grabbing data and playing on my laptop, then I jumped to a server with high cores and a NAS.