Exploring the Limits of Large Language Models as Quant Traders

LLM indeed can replace average human being.

What, so they're better at my hobbies than me? Someone give Claude a 3d printer!

No.

> The models engage in mid-to-low frequency trading (MLFT) trading, where decisions are spaced by minutes to a few hours, not microseconds. In stark contrast to high-frequency trading, MLFT gets us closer to the question we care about: can a model make good choices with a reasonable amount of time and information?

Disagree. Go and chess are games with very limited rules. Succesful trading on the other hand is not so much a arbitary numbers game, but involves analyzing events in the news happening right now. Agentic LLMs that do this and accordingly buy and sell might succeed here.

(Not what they did here, though

"For the first season, they are not given news or access to the leading “narratives” of the market.")

since they're so general, you need to explore if and how you can use them in your domain. guessing 'they're poorly suited' is just that, guessing. in particular:

> We also found that the models were highly sensitive to seemingly trivial prompt changes

this is as much as obvious for anyone who seriously looked at deploying these, that's why there are some very successful startups in the evals space.

Edit - additional detail: The original Asirra paper from October 2007 claimed "Barring a major advance in machine vision, we expect computers will have no better than a 1/54,000 chance of solving it" [0]. It took Philippe Golle from Palo Alto a bit under a year to get "a classifier which is 82.7% accurate in telling apart the images of cats and dogs used in Asirra" and "solve a 12-image Asirra challenge automatically with probability 10.3%" [1].

Edit 2: History is chock-full of examples of human ingenuity solving problems for very little external gain. And here we have a problem where the incentive is almost literally a money printing machine. I expect progress to be very rapid.

[0] https://www.microsoft.com/en-us/research/publication/asirra-...

[1] https://xenon.stanford.edu/~pgolle/papers/dogcat.pdf

But, I havent tested it so far since I do not believe it either :D

But I still think the experiment is interesting because it gives us insight into how LLMs approach risk management, and what effects on that we can have with prompting.

https://www.reddit.com/r/ClaudePlaysPokemon/comments/1otd4kl...

seems like the big issue is just spending time with the tooling to interact with Pokemon and just that calling an LLM for each button is time consuming.

also the other curious nature of the markets is its ability to destroy any persistent trading system by reverting to its core stochastic properties and its constant ebb and flow from stability to instability that crescendos into systematic instability that rewrite the rules all over again.

ive tried all sorts of ways to do this and without being a large institution and being able to absorb the noise for neutral or legal quasi insider trading via proximity, for the average joe the emotional/psychological hardness you need to survive and be in the <1% of traders is simply too much, its not unlike any other sports or arts, many dream the dream but only few get interviewed and written about.

rather i think to myself the best trade is the simplest one: buy shares or invest in a business with money or time (strongly recommend against using this unless you have no other means) and sell it at a higher price or maintain a long term DCF from a business you own as leverage/collateral to arbitrage whatever rate your central bank sets on assets in demand or will be in demand.

to me its clear where LLM fits and doesn't but ultimately it cannot, will not, must not replace your own agency.

(And I'm fairly sure it would be pretty easy to build a system that uses an LLM and a few other small components to beat Pokemon Red. The experiment you are talking about is deliberately hobbled by using a stock LLM without any such tools to make the whole thing entertaining. But when you are trading, you'd want to give your LLM as much help as possible.)

Language can do poker, trading, and other intelligent activities.

The big Quant hedge funds have been using machine learning for decades. I took the coursera RL in finance class years ago.

The idea you are going to beat Two Sigma at their own game with tokens is just an absurdity.

Personally, I think any individual on their own that claims they are doing anything in the algorithmic / ML high frequency space is full of shit.

I could talk like I am too and sound really impressive to someone outside the space. That is much different though than actually making money on what you claim you are doing.

It reminds me of an artist friend when I was younger. She was an artist and I quite liked her paintings. She would tell everyone she is an artist. She was also an encyclopedia when it came to anything art related. She wasn't actually selling much art though. She lived off the $10k a month allowance her rich father gave her. She wasn't even being dishonest but when you didn't know the full picture a person would just assume she was living off her art sales.

I'm honestly more hopeful about AI replacing this process than the core algorithmic component, at least directly. (AI could help write the latter. But it's immediately useful for the former.)

Well, most of them - that can be illegal.

That's probably good news for us index fund investors. We need people to believe they're going to beat the market.

haha, if it would be that easy, most of them would do this? :-D

The thing is - its fucking complicated and most people will give up far before they enter any level of operational capability.

I've developed such a system for myself and Im running it in production (though, not with crypto): And whilte most people will see the complexity in "whatever trading magic you apply", its QUITE the opposite:

- the trading logic itself is simple, its ~ 300 lines

- whats not simple is the part of everything else in the context of "asset management", you need position tracking, state management (orders and positions and account etc.), you need to be able to pour in whatever new quotedata for whatever new assete you identify, the system needs to be stable to work in "mass mode" and be super robust as data provider quality is volatile; you need some type of accounting logic on your side; you need a very capable reporting engine (imagine managing 200 positions simultaneously), I could enlength this list more or less unlimited.

There is MUCH MORE in such an application than the question of "when and how do I trade" - my systems raw source is around 2 MB by today, 3rd party libs and OSS libs not included.

No amount of added context or instructions seems to fix these kind of issues in a way that doesn't still feel pretty hobbled. The only way to get the full power out of the model is to conform your problem to the expectations that seem to be baked in - i.e. just change your rendering coordinate system to be z-up.