1. Backtesting doesn’t mean very much. For lots of reasons real trading is different than backtesting.

2. 8 months is an incredibly short trading window. I care where the market will be in 8 years way more then 8 months.

It seems like back-testing an LLM is going to require significant white-washing of the test data to prevent the LLM from just trading on historical trends it is aware of.

Scrubbing symbol names wouldn't even be enough because I suspect some of these LLMs could "figure out" which stock is, say NVDA, based on the topology of its performance graph.