Over 500PB of data, wow. Would love to know how and why "statistical models that produce price forecasts for over 50,000 financial instruments worldwide" require that much storage.

▲ guerby 3 days ago | parent | next [-]

If you keep all order book changes for a large number of financial instruments volume adds up quickly.

▲ rickette 3 days ago | parent [-]

Would that kind of data not compress like crazy? Or would they need to keep all that data hot and fast.

▲ guerby 3 days ago | parent | next [-]

From just a single exchange you can reach up to 1 million messages of order book change per second

https://www.nasdaqtrader.com/snippets/inet2.html

   Message Volume  1,684,103,265
   Messages per Second  1,134,640
   Order Volume  871,875,595
   Orders per Second  581,696
   Share Volume  12,814,454,760
   Executions per Second  193,350

Also if you look at equity derivative products which have parameters like type call/put, strike, maturity can be hundreds of financial products for one underlying stock.

I worked in this sector and volume of data is a real challenge, no wonder you often get custom software to handle that :)

	▲	rickette 3 days ago \| parent [-]
		Thanks for the insight!

▲ pcthrowaway 2 days ago | parent | prev [-]

How do you propose lossless compression for all orderbook data? Of course if you are willing to lose granularity/information, it can be compressed a lot

▲ Beijinger 3 days ago | parent | prev [-]

Me too. Is is really hard for me to understand, what XTX is actually doing. Trading? VC? AI/ML?

Have you seen their portfolio?

PS: Company seems legit. Impressive growth. But I still don't understand what they are doing. Provide "electronic liquidity". Well....

	▲	orbifold 3 days ago \| parent \| next [-]
		computing correlations between 50.000 financial instruments (X^T X) and doing linear regression ;).
	▲	adastra22 3 days ago \| parent \| prev [-]
		High frequency trading.